{"action":"opened","issue":{"url":"https://api.github.com/repos/CodisLabs/codis/issues/1422","repository_url":"https://api.github.com/repos/CodisLabs/codis","labels_url":"https://api.github.com/repos/CodisLabs/codis/issues/1422/labels{/name}","comments_url":"https://api.github.com/repos/CodisLabs/codis/issues/1422/comments","events_url":"https://api.github.com/repos/CodisLabs/codis/issues/1422/events","html_url":"https://github.com/CodisLabs/codis/issues/1422","id":284432482,"number":1422,"title":"诡异的Read time out！","user":{"login":"jackiechan1981","id":29398311,"avatar_url":"https://avatars3.githubusercontent.com/u/29398311?v=4","gravatar_id":"","url":"https://api.github.com/users/jackiechan1981","html_url":"https://github.com/jackiechan1981","followers_url":"https://api.github.com/users/jackiechan1981/followers","following_url":"https://api.github.com/users/jackiechan1981/following{/other_user}","gists_url":"https://api.github.com/users/jackiechan1981/gists{/gist_id}","starred_url":"https://api.github.com/users/jackiechan1981/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jackiechan1981/subscriptions","organizations_url":"https://api.github.com/users/jackiechan1981/orgs","repos_url":"https://api.github.com/users/jackiechan1981/repos","events_url":"https://api.github.com/users/jackiechan1981/events{/privacy}","received_events_url":"https://api.github.com/users/jackiechan1981/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-12-25T10:08:57Z","updated_at":"2017-12-25T10:08:57Z","closed_at":null,"author_association":"NONE","body":"@spinlock  您好，最近生产环境频发客户端请求Codis的响应超时Read time out，以下是应用的客户端异常信息（Codis3.2，proxy：3个节点-6核-8G，codis-server：6主6从-主节点不持久化-从节点只开RDB-默认save策略，应用配置的响应超时时间是2s）：\r\n`redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out\r\n        at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:202) ~[jedis-2.9.0.jar:na]\r\n        at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.9.0.jar:na]\r\n        at redis.clients.jedis.Protocol.process(Protocol.java:151) ~[jedis-2.9.0.jar:na]\r\n        at redis.clients.jedis.Protocol.read(Protocol.java:215) ~[jedis-2.9.0.jar:na]\r\n        at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340) ~[jedis-2.9.0.jar:na]\r\n        at redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:259) ~[jedis-2.9.0.jar:na]\r\n        at redis.clients.jedis.BinaryJedis.get(BinaryJedis.java:244) ~[jedis-2.9.0.jar:na]\r\nCaused by: java.net.SocketTimeoutException: Read timed out\r\n        at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_51]\r\n        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[na:1.8.0_51]\r\n        at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[na:1.8.0_51]\r\n        at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_51]\r\n        at java.net.SocketInputStream.read(SocketInputStream.java:127) ~[na:1.8.0_51]\r\n        at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:196) ~[jedis-2.9.0.jar:na]\r\n        ... 88 common frames omitted\r\n`\r\n以下是我的分析思路：\r\n1、慢查询堵塞：查看每个codis-server主节点的慢查询记录，在应用Read time out异常的时间点，无慢查询记录；\r\n2、大key的查询造成网络堵塞：zabbix上监控了每个codis-server节点和proxy节点的网卡流量，codis-server不到10M/s，proxy不到30M/s，内网千兆网卡，可排除网络堵塞；\r\n3、使用pipeline造成的缓冲区堵塞：排查业务代码，没有用到pipeline的机制；\r\n4、注：发生Read time out的时间不规律，一般是下午13:00-17:00左右，每次持续1s钟左右，发生异常时，所有使用到此codis集群的应用都同时拋异常；\r\n\r\n以下是最新的监控图表，各种指标跟异常时间点比都变化不大：\r\n![default](https://user-images.githubusercontent.com/29398311/34337858-ff836880-e99d-11e7-9c2f-1cab1aa5fa96.png)\r\n![default](https://user-images.githubusercontent.com/29398311/34337926-8b868650-e99e-11e7-867f-6c6f85fe1606.png)\r\n\r\n@spinlock 不胜感激！"}}
{"actor":{"display_login":"jackiechan1981"}}