title: MySQL Group Replication: Timeout while waiting for the group communication engine to be ready
body: <p>I am running into an issue joining the second instance into a MySQL replication group. It appears to be either a communication or timeout error. In short, I'm seeing this error in the log of the second node:</p>
  
  <pre><code>"Timeout while waiting for the group communication engine to be ready!"
  "The group communication engine is not ready for the member to join. Local port: 33061"
  </code></pre>
  
  <p>The steps I have taken to set up the group:</p>
  
  <h2>1) First I ensured that the proper ports are open between the two servers:</h2>
  
  <p>For reference, I'm using these four variables for privacy:</p>
  
  <ul>
  <li>AA.AAA.AA.A = My FIRST server's PUBLIC IP address </li>
  <li>AA.0.0.000 = My FIRST server's PRIVATE IP address </li>
  <li>BBB.BBB.BBB.BBB = My SECOND server's PUBLIC IP address </li>
  <li>BBB.0.0.000 = My SECOND server's PRIVATE IP address</li>
  </ul>
  
  <p>The FIRST server lives on Amazon AWS, and it has a security group which opens itself to BBB.BBB.BBB.BBB on 3306 and 33061.</p>
  
  <p>The SECOND server lives on a private server and it has a firewall which allows traffic from AA.AAA.AA.A on 3306 and 33061.</p>
  
  <p><strong>From the FIRST server, make sure I can access the second server:</strong></p>
  
  <pre><code>mysql -u repl -p'XXXXXXXXXXXXX' -h BBB.BBB.BBB.BBB
  telnet BBB.BBB.BBB.BBB 3306
  telnet BBB.BBB.BBB.BBB 33061
  telnet BBB.BBB.BBB.BBB 7777 #Times out, as expected
  </code></pre>
  
  <p><strong>From the FIRST server, make sure I can make outgoing connections:</strong></p>
  
  <pre><code>curl portquiz.net:3306
  curl portquiz.net:33061
  </code></pre>
  
  <p><strong>From the SECOND server, make sure I an access the second server:</strong></p>
  
  <pre><code>mysql -u repl -p'XXXXXXXXXXXXX' -h AA.AAA.AA.A
  telnet AA.AAA.AA.A 3306
  telnet AA.AAA.AA.A 33061
  telnet AA.AAA.AA.A 7777 #Times out, as expected
  </code></pre>
  
  <p><strong>From the SECOND server, make sure I can make outgoing connections:</strong></p>
  
  <pre><code>curl portquiz.net:3306
  curl portquiz.net:33061
  </code></pre>
  
  <p>This all seems to be in working order!</p>
  
  <h2>2) Next I start up MySQL on each server with the following configs:</h2>
  
  <p>My FIRST server's Config file:</p>
  
  <pre><code>[mysqld_safe]
  nice                                    = 0
  socket                                  = /var/run/mysqld/mysqld.sock
  
  [mysqld]
  basedir                                 = /usr
  bind_address                            = 0.0.0.0
  binlog_checksum                         = NONE
  binlog_format                           = ROW
  datadir                                 = /var/lib/mysql
  enforce_gtid_consistency                = ON
  general_log                             = 1
  general_log_file                        = /var/log/mysql/mysql.log
  gtid_mode                               = ON
  key_buffer_size                         = 8388608
  lc_messages_dir                         = /usr/share/mysql
  log_bin                                 = binlog
  log_error                               = /var/log/mysql/error.log
  log_slave_updates                       = ON
  long_query_time                         = 60
  loose-group_replication_allow_local_disjoint_gtids_join= ON
  loose-group_replication_bootstrap_group = OFF
  loose-group_replication_enforce_update_everywhere_checks= ON
  loose-group_replication_group_name      = ZZZZZZ-ZZZZZZ-ZZZZZZ-ZZZZZZ
  loose-group_replication_group_seeds     = AA.0.0.000:33061,BBB.BBB.BBB.BBB:33061
  loose-group_replication_ip_whitelist    = AA.0.0.000,BBB.BBB.BBB.BBB,BBB.0.0.000
  loose-group_replication_local_address   = AA.0.0.000:33061
  loose-group_replication_recovery_use_ssl= 1
  loose-group_replication_single_primary_mode= OFF
  loose-group_replication_ssl_mode        = REQUIRED
  loose-group_replication_start_on_boot   = OFF
  master_info_repository                  = TABLE
  max_allowed_packet                      = 16M
  max_binlog_size                         = 100M
  max_connect_errors                      = 100000000
  pid-file                                = /var/run/mysqld/mysqld.pid
  port                                    = 3306
  relay_log                               = first-server-relay-bin
  relay_log_info_repository               = TABLE
  report_host                             = AA.AAA.AA.A
  server_id                               = 1
  skip_external_locking
  slow_query_log                          = 1
  slow_query_log_file                     = /var/log/mysql/slow_query.log
  socket                                  = /var/run/mysqld/mysqld.sock
  thread_cache_size                       = 8
  thread_stack                            = 192K
  tmpdir                                  = /tmp
  transaction_write_set_extraction        = XXHASH64
  user                                    = mysql
  
  [mysqldump]
  max_allowed_packet                      = 16M
  quick
  quote_names
  
  [mysql]
  no_auto_rehash
  
  [isamchk]
  key_buffer_size                         = 16M
  
  [client]
  port                                    = 3306
  socket                                  = /var/run/mysqld/mysqld.sock
  
  !includedir /etc/mysql/conf.d/
  </code></pre>
  
  <p>This is my SECOND server's config:</p>
  
  <pre><code>[mysqld_safe]
  nice                                    = 0
  socket                                  = /var/run/mysqld/mysqld.sock
  
  [mysqld]
  basedir                                 = /usr
  bind_address                            = 0.0.0.0
  binlog_checksum                         = NONE
  binlog_format                           = ROW
  datadir                                 = /var/lib/mysql
  enforce_gtid_consistency                = ON
  general_log                             = 1
  general_log_file                        = /var/log/mysql/mysql.log
  gtid_mode                               = ON
  key_buffer_size                         = 8388608
  lc_messages_dir                         = /usr/share/mysql
  log_bin                                 = binlog
  log_error                               = /var/log/mysql/error.log
  log_slave_updates                       = ON
  long_query_time                         = 60
  loose-group_replication_allow_local_disjoint_gtids_join= ON
  loose-group_replication_bootstrap_group = OFF
  loose-group_replication_enforce_update_everywhere_checks= ON
  loose-group_replication_group_name      = ZZZZZZ-ZZZZZZ-ZZZZZZ-ZZZZZZ
  loose-group_replication_group_seeds     = BBB.0.0.000:33061,AA.AAA.AA.A:33061
  loose-group_replication_ip_whitelist    = BBB.0.0.000,AA.AAA.AA.A,AA.0.0.000
  loose-group_replication_local_address   = BBB.0.0.000:33061
  loose-group_replication_recovery_use_ssl= 1
  loose-group_replication_single_primary_mode= OFF
  loose-group_replication_ssl_mode        = REQUIRED
  loose-group_replication_start_on_boot   = OFF
  master_info_repository                  = TABLE
  max_allowed_packet                      = 16M
  max_binlog_size                         = 100M
  max_connect_errors                      = 100000000
  pid-file                                = /var/run/mysqld/mysqld.pid
  port                                    = 3306
  relay_log                               = second-server-relay-bin
  relay_log_info_repository               = TABLE
  report_host                             = BB.BBB.BB.B
  server_id                               = 2
  skip_external_locking
  slow_query_log                          = 1
  slow_query_log_file                     = /var/log/mysql/slow_query.log
  socket                                  = /var/run/mysqld/mysqld.sock
  thread_cache_size                       = 8
  thread_stack                            = 192K
  tmpdir                                  = /tmp
  transaction_write_set_extraction        = XXHASH64
  user                                    = mysql
  
  [mysqldump]
  max_allowed_packet                      = 16M
  quick
  quote_names
  
  [mysql]
  no_auto_rehash
  
  [isamchk]
  key_buffer_size                         = 16M
  
  [client]
  port                                    = 3306
  socket                                  = /var/run/mysqld/mysqld.sock
  
  !includedir /etc/mysql/conf.d/
  </code></pre>
  
  <p>Both servers start up and there are no errors in the error log.</p>
  
  <h2>3) Now I attempt to bootstrap the groups:</h2>
  
  <p>On the FIRST server:</p>
  
  <pre><code>  mysql&gt; SET GLOBAL GROUP_REPLICATION_BOOTSTRAP_GROUP=ON;
  Query OK, 0 rows affected (0.00 sec)
  
  mysql&gt; START GROUP_REPLICATION;
  Query OK, 0 rows affected (2.11 sec)
  
  mysql&gt; SET GLOBAL GROUP_REPLICATION_BOOTSTRAP_GROUP=OFF;
  Query OK, 0 rows affected (0.00 sec)
  
  mysql&gt; SELECT * FROM performance_schema.replication_group_members;
  +---------------------------+--------------------------------------+-------------+-------------+--------------+
  | CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
  +---------------------------+--------------------------------------+-------------+-------------+--------------+
  | group_replication_applier | 111-111-111-111-111111-111111-111111 | AA.AAA.AA.A |        3306 | ONLINE       |
  +---------------------------+--------------------------------------+-------------+-------------+--------------+
  1 row in set (0.00 sec)
  </code></pre>
  
  <p>This looks good and there are no errors in the log file.</p>
  
  <p>On the SECOND server:</p>
  
  <pre><code>mysql&gt; START GROUP_REPLICATION;
  ERROR 3092 (HY000): The server is not configured properly to be an active member of the group. Please see more details on error log.
  mysql&gt; SELECT * FROM performance_schema.replication_group_members;
  +---------------------------+--------------------------------------+-----------------+-------------+--------------+
  | CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST     | MEMBER_PORT | MEMBER_STATE |
  +---------------------------+--------------------------------------+-----------------+-------------+--------------+
  | group_replication_applier | 222-222-222-222-222222-222222-222222 | BBB.BBB.BBB.BBB |        3306 | OFFLINE      |
  +---------------------------+--------------------------------------+-----------------+-------------+--------------+
  1 row in set (0.00 sec)
  </code></pre>
  
  <p>In the log file, I can see the group replication starting up successfully, the second server connecting to itself (BBB.0.0.000:33061) and the FIRST server (AA.AAA.AA.A:33061) -- "Success connecting using SSL" -- but immediately after I get the following two errors: [GCS] Timeout while waiting for the group communication engine to be ready! and [GCS] The group communication engine is not ready for the member to join. Local port: 33061</p>
  
  <pre><code>2018-04-15T00:15:05.987696Z 5 [Note] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='&lt;NULL&gt;', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='&lt;NULL&gt;', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
  2018-04-15T00:15:05.990639Z 3 [Note] Plugin group_replication reported: 'Group Replication applier module successfully initialized!'
  2018-04-15T00:15:05.990714Z 3 [Note] Plugin group_replication reported: 'auto_increment_increment is set to 7'
  2018-04-15T00:15:05.990754Z 3 [Note] Plugin group_replication reported: 'auto_increment_offset is set to 1'
  2018-04-15T00:15:05.990866Z 8 [Note] Slave SQL thread for channel 'group_replication_applier' initialized, starting replication in log 'FIRST' at position 0, relay log './second-server-relay-bin-group_replication_applier.000012' position: 4
  2018-04-15T00:15:05.991232Z 0 [Note] Plugin group_replication reported: 'state 0 action xa_init'
  2018-04-15T00:15:06.013709Z 0 [Note] Plugin group_replication reported: 'Successfully bound to 0.0.0.0:33061 (socket=51).'
  2018-04-15T00:15:06.013787Z 0 [Note] Plugin group_replication reported: 'Successfully set listen backlog to 32 (socket=51)!'
  2018-04-15T00:15:06.013827Z 0 [Note] Plugin group_replication reported: 'Successfully unblocked socket (socket=51)!'
  2018-04-15T00:15:06.013876Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'
  2018-04-15T00:15:06.013989Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 52'
  2018-04-15T00:15:06.014094Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'
  2018-04-15T00:15:06.014188Z 0 [Note] Plugin group_replication reported: 'Ready to accept incoming connections on 0.0.0.0:33061 (socket=51)!'
  2018-04-15T00:15:06.027381Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'
  2018-04-15T00:15:06.027583Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'
  2018-04-15T00:15:06.027665Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 62'
  2018-04-15T00:15:06.027730Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'
  2018-04-15T00:15:06.040561Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'
  2018-04-15T00:15:06.040752Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'
  2018-04-15T00:15:06.040831Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 66'
  2018-04-15T00:15:06.040894Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'
  2018-04-15T00:15:06.053675Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'
  2018-04-15T00:15:06.053889Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'
  2018-04-15T00:15:06.053975Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 70'
  2018-04-15T00:15:06.054045Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'
  2018-04-15T00:15:06.067043Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'
  2018-04-15T00:15:06.067228Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'
  2018-04-15T00:15:06.067305Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 74'
  2018-04-15T00:15:06.067368Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'
  2018-04-15T00:15:06.080143Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'
  2018-04-15T00:15:06.080326Z 0 [Note] Plugin group_replication reported: 'connecting to BBB.0.0.000 33061'
  2018-04-15T00:15:06.080403Z 0 [Note] Plugin group_replication reported: 'client connected to BBB.0.0.000 33061 fd 78'
  2018-04-15T00:15:06.080465Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'
  2018-04-15T00:15:06.093262Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'
  2018-04-15T00:15:06.093460Z 0 [Note] Plugin group_replication reported: 'connecting to AA.AAA.AA.A 33061'
  2018-04-15T00:15:06.125679Z 0 [Note] Plugin group_replication reported: 'client connected to AA.AAA.AA.A 33061 fd 82'
  2018-04-15T00:15:06.125774Z 0 [Note] Plugin group_replication reported: 'Trying to connect using SSL.'
  2018-04-15T00:15:06.204717Z 0 [Note] Plugin group_replication reported: 'Success connecting using SSL.'
  2018-04-15T00:15:36.270721Z 0 [ERROR] Plugin group_replication reported: '[GCS] Timeout while waiting for the group communication engine to be ready!'
  2018-04-15T00:15:36.270933Z 0 [ERROR] Plugin group_replication reported: '[GCS] The group communication engine is not ready for the member to join. Local port: 33061'
  2018-04-15T00:15:36.271093Z 0 [Note] Plugin group_replication reported: 'state 4337 action xa_terminate'
  2018-04-15T00:15:36.271158Z 0 [Note] Plugin group_replication reported: 'new state x_start'
  2018-04-15T00:15:36.271210Z 0 [Note] Plugin group_replication reported: 'state 4337 action xa_exit'
  2018-04-15T00:15:36.271331Z 0 [Note] Plugin group_replication reported: 'Exiting xcom thread'
  2018-04-15T00:15:36.271383Z 0 [Note] Plugin group_replication reported: 'new state x_start'
  2018-04-15T00:15:36.273406Z 0 [Warning] Plugin group_replication reported: 'read failed'
  2018-04-15T00:15:36.281153Z 0 [ERROR] Plugin group_replication reported: '[GCS] The member was unable to join the group. Local port: 33061'
  2018-04-15T00:16:05.990871Z 3 [ERROR] Plugin group_replication reported: 'Timeout on wait for view after joining group'
  2018-04-15T00:16:05.991036Z 3 [Note] Plugin group_replication reported: 'Requesting to leave the group despite of not being a member'
  2018-04-15T00:16:05.991118Z 3 [ERROR] Plugin group_replication reported: '[GCS] The member is leaving a group without being on one.'
  2018-04-15T00:16:05.991182Z 0 [Note] Plugin group_replication reported: 'Destroying SSL'
  2018-04-15T00:16:05.991242Z 0 [Note] Plugin group_replication reported: 'Success destroying SSL'
  2018-04-15T00:16:05.991352Z 3 [Note] Plugin group_replication reported: 'auto_increment_increment is reset to 1'
  2018-04-15T00:16:05.991408Z 3 [Note] Plugin group_replication reported: 'auto_increment_offset is reset to 1'
  2018-04-15T00:16:05.991618Z 8 [Note] Error reading relay log event for channel 'group_replication_applier': slave SQL thread was killed
  2018-04-15T00:16:05.992230Z 5 [Note] Plugin group_replication reported: 'The group replication applier thread was killed'
  </code></pre>
  
  <p>When I go back to the FIRST server, there is no movement in the logs, they are exactly the same as they were previously. </p>
  
  <p>I can't find any useful information about these two errors: <strong>"Timeout while waiting for the group communication engine to be ready!" and "The group communication engine is not ready for the member to join. Local port: 33061"</strong></p>
  
  <p>Has anyone seen this before and know what it indicates?</p>