{"action":"created","issue":{"url":"https://api.github.com/repos/FriendsOfPHP/Goutte/issues/315","repository_url":"https://api.github.com/repos/FriendsOfPHP/Goutte","labels_url":"https://api.github.com/repos/FriendsOfPHP/Goutte/issues/315/labels{/name}","comments_url":"https://api.github.com/repos/FriendsOfPHP/Goutte/issues/315/comments","events_url":"https://api.github.com/repos/FriendsOfPHP/Goutte/issues/315/events","html_url":"https://github.com/FriendsOfPHP/Goutte/issues/315","id":257561736,"number":315,"title":"<!DOCTYPE html> breaks scraping","user":{"login":"gmask","id":6797513,"avatar_url":"https://avatars1.githubusercontent.com/u/6797513?v=4","gravatar_id":"","url":"https://api.github.com/users/gmask","html_url":"https://github.com/gmask","followers_url":"https://api.github.com/users/gmask/followers","following_url":"https://api.github.com/users/gmask/following{/other_user}","gists_url":"https://api.github.com/users/gmask/gists{/gist_id}","starred_url":"https://api.github.com/users/gmask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gmask/subscriptions","organizations_url":"https://api.github.com/users/gmask/orgs","repos_url":"https://api.github.com/users/gmask/repos","events_url":"https://api.github.com/users/gmask/events{/privacy}","received_events_url":"https://api.github.com/users/gmask/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-09-14T00:43:28Z","updated_at":"2017-12-20T17:35:39Z","closed_at":null,"author_association":"NONE","body":"Hey there, I'm attempting to scrape IMDb, but the very first line of their site breaks the scraper: `<!DOCTYPE html>`\r\n\r\nSaving the file locally and removing that one line fixes the issue and Goutte returns the data I want.\r\n\r\n*Update*\r\nWell, I guess it's not that tag per se because many other sites that declare identically work just fine, but for some reason removing it makes IMDb work. So, there's something elsewhere that the site is doing that breaks Goutte, but I have no idea what it could be.\r\n\r\n*Update 2*\r\nOk, so I've traced to the following: removing the `DOCTYPE` declaration allows the entire page to load, which means scraping works. Adding it prevents Goutte from fetching anymore content outside of `<head>`. When I look at the loaded HTML with the proper HTML and `DOCTYPE` only anything within the `<head>` tags appears, making `</head>` the end of the file.\r\n\r\n*Update 3*\r\nFound the culprit: IMDb uses two javascript ads that completely overwrite the document's head property depending on the ad being loaded. Therefore there are 3 doctype declarations on the page which breaks Goutte. Removing the <script> from the DOM fixes the issue completely, however I cannot do this with the live site. Is there a way to prevent certain scripts from loading, or otherwise deal with this issue?"},"comment":{"url":"https://api.github.com/repos/FriendsOfPHP/Goutte/issues/comments/353130739","html_url":"https://github.com/FriendsOfPHP/Goutte/issues/315#issuecomment-353130739","issue_url":"https://api.github.com/repos/FriendsOfPHP/Goutte/issues/315","id":353130739,"user":{"login":"gmask","id":6797513,"avatar_url":"https://avatars1.githubusercontent.com/u/6797513?v=4","gravatar_id":"","url":"https://api.github.com/users/gmask","html_url":"https://github.com/gmask","followers_url":"https://api.github.com/users/gmask/followers","following_url":"https://api.github.com/users/gmask/following{/other_user}","gists_url":"https://api.github.com/users/gmask/gists{/gist_id}","starred_url":"https://api.github.com/users/gmask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gmask/subscriptions","organizations_url":"https://api.github.com/users/gmask/orgs","repos_url":"https://api.github.com/users/gmask/repos","events_url":"https://api.github.com/users/gmask/events{/privacy}","received_events_url":"https://api.github.com/users/gmask/received_events","type":"User","site_admin":false},"created_at":"2017-12-20T17:35:39Z","updated_at":"2017-12-20T17:35:39Z","author_association":"NONE","body":"@stof Do you know of a way to force BrowserKit to ignore everything inside `<script>` tags?"}}
{"actor":{"display_login":"gmask"}}