title: Elk Logstash not able to connect to Event Hub Azure
body: <p>I am trying to connect to Azure Event hub from ElasticSearch logstash. Here is the details of azure module I mentioned in /etc/logstash/logstash.yml file.</p>
  
  <pre><code>modules:
    - name: azure
      var.elasticsearch.hosts: localhost:9200
      var.kibana.host: localhost:5601
      var.input.azure_event_hubs.consumer_group: "default"
      var.input.azure_event_hubs.storage_connection: "DefaultEndpointsProtocol=https;AccountName=ABC;AccountKey=XXXXX==;EndpointSuffix=core.windows.net"
      var.input.azure_event_hubs.threads: 9
      var.input.azure_event_hubs.event_hub_connections:
        - "Endpoint=sb://elk-event.servicebus.windows.net/;SharedAccessKeyName=share-log;SharedAccessKey=YYYYY=;EntityPath=elk-log-event"
  
  </code></pre>
  
  <p>But from when I launch logstash using command:</p>
  
  <pre><code>./logstash --path.settings=/etc/logstash
  </code></pre>
  
  <p>I used consumer_group as logstash as well as default. Is it an issue?</p>
  
  <p>it throws error as:</p>
  
  <pre><code>[2019-10-04T09:43:46,332][ERROR][com.microsoft.azure.eventprocessorhost.PartitionPump] host logstash-8331671e-0dea-4f8d-88de-ba3ebc974dca: 1: PartitionReceiver creation failed
  java.util.concurrent.CompletionException: com.microsoft.azure.eventhubs.IllegalEntityException: The messaging entity 'elk-event:eventhub:elk-log-event~32766|default' could not be found. TrackingId:c380d842-e946-4f12-ba12-daecdf60df2d_B29, SystemTracker:elk-event:eventhub:elk-log-event~32766|default, Timestamp:2019-10-04T09:43:45 TrackingId:778a3beb42734bc098c7419f5f4b8c21_G5, SystemTracker:gateway5, Timestamp:2019-10-04T09:43:45, errorContext[NS: elk-event.servicebus.windows.net, PATH: elk-log-event/ConsumerGroups/default/Partitions/1, REFERENCE_ID: e393e0_8c21_G5_1570182225737]
          at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_222]
          at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_222]
          at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:647) ~[?:1.8.0_222]
          at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) ~[?:1.8.0_222]
          at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_222]
          at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_222]
          at com.microsoft.azure.eventhubs.impl.ExceptionUtil.completeExceptionally(ExceptionUtil.java:98) ~[azure-eventhubs-2.2.0.jar:?]
          at com.microsoft.azure.eventhubs.impl.MessageReceiver.cancelOpen(MessageReceiver.java:357) ~[azure-eventhubs-2.2.0.jar:?]
          at com.microsoft.azure.eventhubs.impl.MessageReceiver.onOpenComplete(MessageReceiver.java:347) ~[azure-eventhubs-2.2.0.jar:?]
          at com.microsoft.azure.eventhubs.impl.MessageReceiver.onError(MessageReceiver.java:414) ~[azure-eventhubs-2.2.0.jar:?]
          at com.microsoft.azure.eventhubs.impl.MessageReceiver.onClose(MessageReceiver.java:736) ~[azure-eventhubs-2.2.0.jar:?]
          at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.processOnClose(BaseLinkHandler.java:74) ~[azure-eventhubs-2.2.0.jar:?]
          at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.handleRemoteLinkClosed(BaseLinkHandler.java:113) ~[azure-eventhubs-2.2.0.jar:?]
          at com.microsoft.azure.eventhubs.impl.BaseLinkHandler.onLinkRemoteClose(BaseLinkHandler.java:48) ~[azure-eventhubs-2.2.0.jar:?]
          at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176) ~[proton-j-0.33.0.jar:?]
          at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108) ~[proton-j-0.33.0.jar:?]
          at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324) ~[proton-j-0.33.0.jar:?]
          at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291) ~[proton-j-0.33.0.jar:?]
          at com.microsoft.azure.eventhubs.impl.MessagingFactory$RunReactor.run(MessagingFactory.java:507) ~[azure-eventhubs-2.2.0.jar:?]
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222]
          at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222]
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222]
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
          at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
  Caused by: com.microsoft.azure.eventhubs.IllegalEntityException: The messaging entity 'elk-event:eventhub:elk-log-event~32766|default' could not be found. TrackingId:c380d842-e946-4f12-ba12-daecdf60df2d_B29, SystemTracker:elk-event:eventhub:elk-log-event~32766|default, Timestamp:2019-10-04T09:43:45 TrackingId:778a3beb42734bc098c7419f5f4b8c21_G5, SystemTracker:gateway5, Timestamp:2019-10-04T09:43:45, errorContext[NS: elk-event.servicebus.windows.net, PATH: elk-log-event/ConsumerGroups/default/Partitions/1, REFERENCE_ID: e393e0_8c21_G5_1570182225737]
          at com.microsoft.azure.eventhubs.impl.ExceptionUtil.distinguishNotFound(ExceptionUtil.java:86) ~[azure-eventhubs-2.2.0.jar:?]
          at com.microsoft.azure.eventhubs.impl.ExceptionUtil.toException(ExceptionUtil.java:31) ~[azure-eventhubs-2.2.0.jar:?]
          at com.microsoft.azure.eventhubs.impl.MessageReceiver.onClose(MessageReceiver.java:735) ~[azure-eventhubs-2.2.0.jar:?]
          ... 15 more
  </code></pre>
  
  <p>Is it related with consumer group or something else? If so, I am not sure how to create my own consumer group. </p>
  
  <p>Any feedback? Thanks in advance.</p>