{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-java/issues/2722","repository_url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-java","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-java/issues/2722/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-java/issues/2722/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-java/issues/2722/events","html_url":"https://github.com/GoogleCloudPlatform/google-cloud-java/issues/2722","id":283050429,"number":2722,"title":"Pub/Sub: publisher fails to publish more than 1000 messages, with DEADLINE_EXCEEDED","user":{"login":"kir-titievsky","id":18533398,"avatar_url":"https://avatars0.githubusercontent.com/u/18533398?v=4","gravatar_id":"","url":"https://api.github.com/users/kir-titievsky","html_url":"https://github.com/kir-titievsky","followers_url":"https://api.github.com/users/kir-titievsky/followers","following_url":"https://api.github.com/users/kir-titievsky/following{/other_user}","gists_url":"https://api.github.com/users/kir-titievsky/gists{/gist_id}","starred_url":"https://api.github.com/users/kir-titievsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kir-titievsky/subscriptions","organizations_url":"https://api.github.com/users/kir-titievsky/orgs","repos_url":"https://api.github.com/users/kir-titievsky/repos","events_url":"https://api.github.com/users/kir-titievsky/events{/privacy}","received_events_url":"https://api.github.com/users/kir-titievsky/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-12-18T23:03:24Z","updated_at":"2017-12-19T00:07:51Z","closed_at":null,"author_association":"NONE","body":"I've slightly modified this very basic publisher code offered to me by one of the customers. The code runs when we try to publish 1000 messages, but fails with a gRPC timeout exception when I try to run it with 10K messages.    \r\n<pre>\r\nimport com.codahale.metrics.Counter;\r\nimport com.codahale.metrics.MetricRegistry;\r\nimport com.google.api.core.ApiFuture;\r\nimport com.google.api.core.ApiFutureCallback;\r\nimport com.google.api.core.ApiFutures;\r\nimport com.google.cloud.ServiceOptions;\r\nimport com.google.cloud.pubsub.v1.Publisher;\r\nimport com.google.protobuf.ByteString;\r\nimport com.google.pubsub.v1.PubsubMessage;\r\nimport com.google.pubsub.v1.TopicName;\r\n\r\n\r\nimport java.util.Random;\r\n\r\npublic class App {\r\n    public static void main(String[] args) throws Exception {\r\n        MetricRegistry metricRegistry = new MetricRegistry();\r\n        Counter failedPubSubSends = metricRegistry.counter(\"fail\");\r\n        Counter successfulPubSubSends = metricRegistry.counter(\"success\");\r\n        TopicName topicName = TopicName.of(ServiceOptions.getDefaultProjectId(), args[0]);\r\n        Publisher.Builder builder = Publisher.newBuilder(topicName);\r\n        Publisher publisher = builder.build();\r\n        long millis = System.currentTimeMillis();\r\n        Integer numberOfMessages = Integer.valueOf(args[1   ]);\r\n        try {\r\n            byte[] data = new byte[700];\r\n            new Random().nextBytes(data);\r\n            for (int i = 0; i < numberOfMessages; i++) {\r\n                ApiFuture<String> resultFuture = publisher.publish(PubsubMessage.newBuilder().setData(ByteString.copyFrom(data)).build());\r\n                ApiFutures.addCallback(resultFuture, new ApiFutureCallback<String>() {\r\n                    public void onSuccess(String result) {\r\n                        successfulPubSubSends.inc();\r\n                    }\r\n                    public void onFailure(Throwable t) {\r\n                        failedPubSubSends.inc();\r\n                        System.out.println(t.toString());\r\n                    }\r\n                });\r\n            }\r\n        } finally {\r\n            publisher.shutdown();\r\n            System.out.println(\"took \" + (System.currentTimeMillis() - millis) + \" ms for \" + numberOfMessages\r\n                    + \" (success \" + successfulPubSubSends.getCount() + \" / failed \" + failedPubSubSends.getCount() + \")\");\r\n        }\r\n    }\r\n}\r\n</pre>\r\nThe output is mainly lines like this:\r\n<pre> \r\ncom.google.api.gax.rpc.DeadlineExceededException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 9999990738ns\r\n</pre>\r\nAnd the final outcome is a typical 80-90% failure rate.\r\n<pre>\r\ntook 16906 ms for 100000 (success 12232 / failed 87768)\r\n</pre>\r\n\r\nServerside, I see no errors which suggests that this is a client side issue. \r\n\r\nWhat I would expect in this case: the code just publishes this set of messages."},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-java/issues/comments/352597992","html_url":"https://github.com/GoogleCloudPlatform/google-cloud-java/issues/2722#issuecomment-352597992","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/google-cloud-java/issues/2722","id":352597992,"user":{"login":"pongad","id":1617025,"avatar_url":"https://avatars1.githubusercontent.com/u/1617025?v=4","gravatar_id":"","url":"https://api.github.com/users/pongad","html_url":"https://github.com/pongad","followers_url":"https://api.github.com/users/pongad/followers","following_url":"https://api.github.com/users/pongad/following{/other_user}","gists_url":"https://api.github.com/users/pongad/gists{/gist_id}","starred_url":"https://api.github.com/users/pongad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pongad/subscriptions","organizations_url":"https://api.github.com/users/pongad/orgs","repos_url":"https://api.github.com/users/pongad/repos","events_url":"https://api.github.com/users/pongad/events{/privacy}","received_events_url":"https://api.github.com/users/pongad/received_events","type":"User","site_admin":false},"created_at":"2017-12-19T00:07:51Z","updated_at":"2017-12-19T00:07:51Z","author_association":"CONTRIBUTOR","body":"The main problem is that we're trying to send messages faster than the network can deliver them.\r\nWe decided a few months ago that we won't implement back-pressure in client lib layer since the use-case is too varied.\r\n\r\nI'll make a documentation PR for this."}}
{"actor":{"display_login":"pongad"}}