title: hazelcast client not getting the full dataset from a cluster
body: <p>I am using hazelcast v3.6 (client-server mode).  The relevant parts of the client and server configs are copied below.  When I call IMap.size(), I get the count for the full set of data that I have inserted into the test cluster (2 node).  However, when I get the keySet or entrySet, I only get the set for half of the keys/objects.  I tried running the localKeySet from the client but that throws the following exception:</p>
  
  <pre><code>Exception in thread "main" java.lang.UnsupportedOperationException: Locality is ambiguous for client!!!
      at com.hazelcast.client.proxy.ClientMapProxy.localKeySet(ClientMapProxy.java:1172)
  </code></pre>
  
  <p>I think localKeySet is not available on the client but need to reconfirm that as well.</p>
  
  <p><strong>Client config</strong> - relevant parts, not the full config:</p>
  
  <pre><code>    &lt;hazelcast-client xsi:schemaLocation="http://www.hazelcast.com/schema/client-config hazelcast-client-config-3.6.xsd"
                        xmlns="http://www.hazelcast.com/schema/client-config"
                        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"&gt;
  
          &lt;properties&gt;
              &lt;property name="hazelcast.client.shuffle.member.list"&gt;true&lt;/property&gt;
              &lt;property name="hazelcast.client.heartbeat.timeout"&gt;60000&lt;/property&gt;
              &lt;property name="hazelcast.client.heartbeat.interval"&gt;5000&lt;/property&gt;
              &lt;property name="hazelcast.client.event.thread.count"&gt;5&lt;/property&gt;
              &lt;property name="hazelcast.client.event.queue.capacity"&gt;1000000&lt;/property&gt;
              &lt;property name="hazelcast.client.invocation.timeout.seconds"&gt;120&lt;/property&gt;
          &lt;/properties&gt;
  
  
          &lt;network&gt;
              &lt;cluster-members&gt;
  &lt;!-- ip addresses replaced --&gt;
                  &lt;address&gt;101.102.103.104&lt;/address&gt;
                  &lt;address&gt;101.102.103.105&lt;/address&gt;
              &lt;/cluster-members&gt;
              &lt;smart-routing&gt;true&lt;/smart-routing&gt;
              &lt;redo-operation&gt;true&lt;/redo-operation&gt;
              &lt;connection-timeout&gt;60000&lt;/connection-timeout&gt;
              &lt;connection-attempt-period&gt;3000&lt;/connection-attempt-period&gt;
              &lt;connection-attempt-limit&gt;2&lt;/connection-attempt-limit&gt;
              &lt;socket-options&gt;
                  &lt;tcp-no-delay&gt;false&lt;/tcp-no-delay&gt;
                  &lt;keep-alive&gt;true&lt;/keep-alive&gt;
                  &lt;reuse-address&gt;true&lt;/reuse-address&gt;
                  &lt;linger-seconds&gt;3&lt;/linger-seconds&gt;
                  &lt;timeout&gt;-1&lt;/timeout&gt;
                  &lt;buffer-size&gt;32&lt;/buffer-size&gt;
              &lt;/socket-options&gt;
          &lt;/network&gt;
  </code></pre>
  
  <p><strong>Server Config</strong> - <em>relevant parts</em></p>
  
  <pre><code>   &lt;network&gt;
          &lt;port auto-increment="true" port-count="100"&gt;5701&lt;/port&gt;
          &lt;outbound-ports&gt;
              &lt;ports&gt;0-5900&lt;/ports&gt;
          &lt;/outbound-ports&gt;
          &lt;join&gt;
              &lt;multicast enabled="false"&gt;
              &lt;/multicast&gt;
              &lt;tcp-ip enabled="true"&gt;
                  &lt;member&gt;101.102.103.104&lt;/member&gt;
                  &lt;member&gt;101.102.103.105&lt;/member&gt;
               &lt;/tcp-ip&gt;
          &lt;/join&gt;
          &lt;interfaces enabled="true"&gt;
              &lt;interface&gt;101.102.103.104&lt;/interface&gt; &lt;!-- this is modified for the second server node --&gt;
          &lt;/interfaces&gt;
          &lt;ssl enabled="false" /&gt;
          &lt;socket-interceptor enabled="false" /&gt;
          &lt;symmetric-encryption enabled="false"&gt;
              &lt;algorithm&gt;PBEWithMD5AndDES&lt;/algorithm&gt;
              &lt;!-- salt value to use when generating the secret key --&gt;
              &lt;salt&gt;test&lt;/salt&gt;
              &lt;!-- pass phrase to use when generating the secret key --&gt;
              &lt;password&gt;testpass&lt;/password&gt;
              &lt;!-- iteration count to use when generating the secret key --&gt;
              &lt;iteration-count&gt;19&lt;/iteration-count&gt;
          &lt;/symmetric-encryption&gt;
      &lt;/network&gt;
  </code></pre>
  
  <p>Any thoughts on what might be causing the issue and ways to fix it?</p>
  
  <p><strong>Update</strong>:
  I have tried replicating the setup on another two node cluster but have not been able to reproduce the same issue.  When I call the keySet on the new cluster, I get the full set of keys, not half (unlike the original cluster).  I am aware that I can use Predicates instead of getting the whole keySet.</p>
  
  <p><strong>Update</strong>
  I will try to do that in the near future - I have to dig through the logs.  The issue was resolved when I brought down one node and the second node became the primary for all the data (it was a two node test cluster with async replication factor of 1).  After that, when I brought up the second node again, the data was distributed again and I did not see any issues with the two node cluster.  So it appears that something happened after the cluster was brought up that resulted in the hazelcast client getting the map values from both nodes but any call to the keyset/entry set was just getting keys/values from one node.  I tried switching the order in the hazelcast client (xml file) to see if that would change the keyset/entries that the client was receiving but that did not solve the issue.  </p>