{"action":"created","issue":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/2943","repository_url":"https://api.github.com/repos/scrapy/scrapy","labels_url":"https://api.github.com/repos/scrapy/scrapy/issues/2943/labels{/name}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/2943/comments","events_url":"https://api.github.com/repos/scrapy/scrapy/issues/2943/events","html_url":"https://github.com/scrapy/scrapy/issues/2943","id":260311303,"number":2943,"title":"Scrapy ImagePipeline ignore images from specific host","user":{"login":"marcuslind90","id":8709266,"avatar_url":"https://avatars0.githubusercontent.com/u/8709266?v=4","gravatar_id":"","url":"https://api.github.com/users/marcuslind90","html_url":"https://github.com/marcuslind90","followers_url":"https://api.github.com/users/marcuslind90/followers","following_url":"https://api.github.com/users/marcuslind90/following{/other_user}","gists_url":"https://api.github.com/users/marcuslind90/gists{/gist_id}","starred_url":"https://api.github.com/users/marcuslind90/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marcuslind90/subscriptions","organizations_url":"https://api.github.com/users/marcuslind90/orgs","repos_url":"https://api.github.com/users/marcuslind90/repos","events_url":"https://api.github.com/users/marcuslind90/events{/privacy}","received_events_url":"https://api.github.com/users/marcuslind90/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-09-25T15:07:19Z","updated_at":"2017-09-26T13:42:10Z","closed_at":"2017-09-26T10:25:18Z","author_association":"NONE","body":"I have an issue where my crawler is ignoring images from a specific host. It does not download any images and it does not throw any error or log messages. If I replace the image path with any other image path from any other source, it works fine with exactly the same config and code.\r\n\r\nI put up a public repo here that reproduce it:\r\nhttps://github.com/marcuslind90/scrapy_error\r\n\r\nAs you can see in the `example` spider, I hardcode 2 paths like this:\r\n\r\n```\r\nloader.add_value('image_urls', ['https://media.fastighetsbyran.se/22943836.jpg', 'http://hemmon.com/house.jpg'])\r\n```\r\n\r\nThe first image is ignored (and any other images on that host) while the second image is downloaded.\r\n\r\nAnyone recognize this issue? Could it have something to do with headers? Also, I believe that the host is programmatically generating the images because you can append a `?Bredd=<pixels>` parameter that will automatically resize the picture. Perhaps this could be part of it as well."},"comment":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/comments/332201944","html_url":"https://github.com/scrapy/scrapy/issues/2943#issuecomment-332201944","issue_url":"https://api.github.com/repos/scrapy/scrapy/issues/2943","id":332201944,"user":{"login":"IAlwaysBeCoding","id":7705309,"avatar_url":"https://avatars1.githubusercontent.com/u/7705309?v=4","gravatar_id":"","url":"https://api.github.com/users/IAlwaysBeCoding","html_url":"https://github.com/IAlwaysBeCoding","followers_url":"https://api.github.com/users/IAlwaysBeCoding/followers","following_url":"https://api.github.com/users/IAlwaysBeCoding/following{/other_user}","gists_url":"https://api.github.com/users/IAlwaysBeCoding/gists{/gist_id}","starred_url":"https://api.github.com/users/IAlwaysBeCoding/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/IAlwaysBeCoding/subscriptions","organizations_url":"https://api.github.com/users/IAlwaysBeCoding/orgs","repos_url":"https://api.github.com/users/IAlwaysBeCoding/repos","events_url":"https://api.github.com/users/IAlwaysBeCoding/events{/privacy}","received_events_url":"https://api.github.com/users/IAlwaysBeCoding/received_events","type":"User","site_admin":false},"created_at":"2017-09-26T13:42:09Z","updated_at":"2017-09-26T13:42:09Z","author_association":"NONE","body":"@marcuslind90  I'm glad that you have figured out your problem. Also, if you check out the [issue](https://github.com/scrapy/scrapy/issues/2371) I pointed out in my previous reply. The @geemaple figured out that robots.txt file at imgs.bipush.com prohibit every scraper that respects the robots.txt from scraping the site.\r\n\r\nPut this on your settings.py\r\n`ROBOTSTXT_OBEYROBOTSTXT_OBEY=False`\r\n\r\nThis will make scrapy ignore robots.txt files and scrape through them. That should fix your problem that you have.\r\n\r\n\r\n"}}
{"actor":{"display_login":"IAlwaysBeCoding"}}