{"action":"created","issue":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/2774","repository_url":"https://api.github.com/repos/scrapy/scrapy","labels_url":"https://api.github.com/repos/scrapy/scrapy/issues/2774/labels{/name}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/2774/comments","events_url":"https://api.github.com/repos/scrapy/scrapy/issues/2774/events","html_url":"https://github.com/scrapy/scrapy/issues/2774","id":233094204,"number":2774,"title":"Don`t fetch url","user":{"login":"tonal","id":316216,"avatar_url":"https://avatars0.githubusercontent.com/u/316216?v=4","gravatar_id":"","url":"https://api.github.com/users/tonal","html_url":"https://github.com/tonal","followers_url":"https://api.github.com/users/tonal/followers","following_url":"https://api.github.com/users/tonal/following{/other_user}","gists_url":"https://api.github.com/users/tonal/gists{/gist_id}","starred_url":"https://api.github.com/users/tonal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tonal/subscriptions","organizations_url":"https://api.github.com/users/tonal/orgs","repos_url":"https://api.github.com/users/tonal/repos","events_url":"https://api.github.com/users/tonal/events{/privacy}","received_events_url":"https://api.github.com/users/tonal/received_events","type":"User","site_admin":false},"labels":[{"id":13907246,"url":"https://api.github.com/repos/scrapy/scrapy/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-06-02T07:02:47Z","updated_at":"2017-07-27T18:21:33Z","closed_at":null,"body":"Url `http://www.tehnosad.ru/subcategory/?id=584&producers[]=524` do not fetch from **scrapy**.\r\nBat **httpie** and **chrome** get Ok.\r\nHow to correct it?\r\n```\r\n$ scrapy fetch --headers 'http://www.tehnosad.ru/subcategory/?id=584&producers[]=524'\r\n2017-06-02 13:49:44 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: remains)\r\n2017-06-02 13:49:44 [scrapy.utils.log] INFO: Overridden settings: {'RETRY_TIMES': 5, 'SPIDER_MODULES': ['remains.spiders'], 'FEED_URI': './%(name)s.csv', 'HTTPCACHE_EXPIRATION_SECS': 600, 'RETRY_HTTP_CODES': [400, 408, 420, 500, 502, 503, 504], 'BOT_NAME': 'remains', 'FEED_FORMAT': 'csv', 'AUTOTHROTTLE_ENABLED': True, 'HTTPCACHE_STORAGE': 'scrapy.contrib.httpcache.FilesystemCacheStorage', 'NEWSPIDER_MODULE': 'remains.spiders', 'HTTPCACHE_DIR': './httpcache', 'DOWNLOADER_CLIENTCONTEXTFACTORY': 'remains.ssl_context.CustomClientContextFactory'}\r\n2017-06-02 13:49:44 [scrapy.middleware] INFO: Enabled extensions:\r\n['scrapy.extensions.logstats.LogStats',\r\n 'scrapy.extensions.feedexport.FeedExporter',\r\n 'scrapy.extensions.memusage.MemoryUsage',\r\n 'scrapy.extensions.corestats.CoreStats',\r\n 'scrapy.extensions.throttle.AutoThrottle']\r\n2017-06-02 13:49:44 [scrapy.middleware] INFO: Enabled downloader middlewares:\r\n['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',\r\n 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',\r\n 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',\r\n 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',\r\n 'scrapy.downloadermiddlewares.retry.RetryMiddleware',\r\n 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',\r\n 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',\r\n 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',\r\n 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',\r\n 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',\r\n 'scrapy.downloadermiddlewares.stats.DownloaderStats']\r\n2017-06-02 13:49:44 [scrapy.middleware] INFO: Enabled spider middlewares:\r\n['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',\r\n 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',\r\n 'scrapy.spidermiddlewares.referer.RefererMiddleware',\r\n 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',\r\n 'scrapy.spidermiddlewares.depth.DepthMiddleware']\r\n2017-06-02 13:49:44 [scrapy.middleware] INFO: Enabled item pipelines:\r\n['remains.pipelines.RemainsPipeline']\r\n2017-06-02 13:49:44 [scrapy.core.engine] INFO: Spider opened\r\n2017-06-02 13:49:44 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)\r\n2017-06-02 13:49:44 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:49:51 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:49:59 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:50:05 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:50:10 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:50:18 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:50:22 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:50:29 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:50:36 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:50:40 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:50:44 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)\r\n2017-06-02 13:50:47 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:50:52 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:50:59 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:51:06 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:51:13 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:51:20 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:51:26 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:51:32 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:51:39 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:51:44 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)\r\n2017-06-02 13:51:47 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524> from <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>\r\n2017-06-02 13:51:54 [scrapy.downloadermiddlewares.redirect] DEBUG: Discarding <GET http://www.tehnosad.ru/subcategory/?id=584&producers%5B%5D=524>: max redirections reached\r\n2017-06-02 13:51:54 [scrapy.core.engine] INFO: Closing spider (finished)\r\n2017-06-02 13:51:54 [scrapy.statscollectors] INFO: Dumping Scrapy stats:\r\n{'downloader/request_bytes': 9196,\r\n 'downloader/request_count': 21,\r\n 'downloader/request_method_count/GET': 21,\r\n 'downloader/response_bytes': 14284,\r\n 'downloader/response_count': 21,\r\n 'downloader/response_status_count/301': 21,\r\n 'finish_reason': 'finished',\r\n 'finish_time': datetime.datetime(2017, 6, 2, 6, 51, 54, 206787),\r\n 'log_count/DEBUG': 21,\r\n 'log_count/INFO': 9,\r\n 'memusage/max': 50880512,\r\n 'memusage/startup': 49733632,\r\n 'scheduler/dequeued': 21,\r\n 'scheduler/dequeued/memory': 21,\r\n 'scheduler/enqueued': 21,\r\n 'scheduler/enqueued/memory': 21,\r\n 'start_time': datetime.datetime(2017, 6, 2, 6, 49, 44, 629809)}\r\n2017-06-02 13:51:54 [scrapy.core.engine] INFO: Spider closed (finished)\r\n```"},"comment":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/comments/318445241","html_url":"https://github.com/scrapy/scrapy/issues/2774#issuecomment-318445241","issue_url":"https://api.github.com/repos/scrapy/scrapy/issues/2774","id":318445241,"user":{"login":"redapple","id":886296,"avatar_url":"https://avatars1.githubusercontent.com/u/886296?v=4","gravatar_id":"","url":"https://api.github.com/users/redapple","html_url":"https://github.com/redapple","followers_url":"https://api.github.com/users/redapple/followers","following_url":"https://api.github.com/users/redapple/following{/other_user}","gists_url":"https://api.github.com/users/redapple/gists{/gist_id}","starred_url":"https://api.github.com/users/redapple/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/redapple/subscriptions","organizations_url":"https://api.github.com/users/redapple/orgs","repos_url":"https://api.github.com/users/redapple/repos","events_url":"https://api.github.com/users/redapple/events{/privacy}","received_events_url":"https://api.github.com/users/redapple/received_events","type":"User","site_admin":false},"created_at":"2017-07-27T18:21:33Z","updated_at":"2017-07-27T18:21:33Z","body":"The issue is fixed when using the current master branch of https://github.com/scrapy/w3lib .\r\nI'll close this when we release w3lib v1.18"}}
{"actor":{"display_login":"redapple"}}