{"action":"created","issue":{"url":"https://api.github.com/repos/openucx/ucx/issues/1760","repository_url":"https://api.github.com/repos/openucx/ucx","labels_url":"https://api.github.com/repos/openucx/ucx/issues/1760/labels{/name}","comments_url":"https://api.github.com/repos/openucx/ucx/issues/1760/comments","events_url":"https://api.github.com/repos/openucx/ucx/issues/1760/events","html_url":"https://github.com/openucx/ucx/issues/1760","id":250247667,"number":1760,"title":"Hang with RC and UD","user":{"login":"amaslenn","id":1412944,"avatar_url":"https://avatars2.githubusercontent.com/u/1412944?v=4","gravatar_id":"","url":"https://api.github.com/users/amaslenn","html_url":"https://github.com/amaslenn","followers_url":"https://api.github.com/users/amaslenn/followers","following_url":"https://api.github.com/users/amaslenn/following{/other_user}","gists_url":"https://api.github.com/users/amaslenn/gists{/gist_id}","starred_url":"https://api.github.com/users/amaslenn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amaslenn/subscriptions","organizations_url":"https://api.github.com/users/amaslenn/orgs","repos_url":"https://api.github.com/users/amaslenn/repos","events_url":"https://api.github.com/users/amaslenn/events{/privacy}","received_events_url":"https://api.github.com/users/amaslenn/received_events","type":"User","site_admin":false},"labels":[{"id":138949041,"url":"https://api.github.com/repos/openucx/ucx/labels/bug","name":"bug","color":"fc2929","default":true},{"id":448762857,"url":"https://api.github.com/repos/openucx/ucx/labels/external","name":"external","color":"fbca04","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-08-15T08:04:05Z","updated_at":"2017-10-01T12:36:12Z","closed_at":"2017-10-01T12:36:12Z","author_association":"CONTRIBUTOR","body":"MOFED: `MLNX_OFED_LINUX-4.1-1.0.2.0`\r\nRun details: `num_nodes=8 ppn=32(x8)`\r\nMTT test log: http://e2e-gw.mellanox.com:4080//mnt/lustre/users/mtt/scratch/ucx_ompi/20170814_230941_24122_19168_clx-hercules-069/html/test_stdout_LE2OOS.txt\r\n\r\nWith `TLS=rc,sm` test hangs with enabled HW TM only. (works OK with `NP=240`)\r\nWith `TLS=ud,sm` test hangs. (works OK with `NP=8`)\r\nWith `TLS=rc_x,sm` no hang.\r\nWith `TLS=dc_x,sm` no hang.\r\n\r\n\r\nOriginal cmd (`NP=256`):\r\n`/hpc/local/benchmarks/hpcx_install_Monday/hpcx-gcc-redhat7.2/ompi-v3.0.x/bin/mpirun -np 256 -mca btl_openib_warn_default_gid_prefix 0 --debug-daemons --bind-to core --tag-output --timestamp-output --display-map -mca pml ucx -x UCX_NET_DEVICES=mlx5_2:1 -mca btl_openib_if_include mlx5_2:1 -mca coll_hcoll_enable 0 -x UCX_IB_GID_INDEX=0 -x UCX_TLS=rc,sm -x UCX_RC_VERBS_TM_ENABLE=y --map-by node /mnt/lustre/users/mtt/scratch/ucx_ompi/20170814_230941_24122_19168_clx-hercules-069/installs/_kUV/tests/mpich_tests/mpich-mellanox.git/test/mpi/coll/nbicallgather`\r\n\r\nTrace example for original cmd:\r\n```sh\r\nThread 4 (Thread 0x7ffff4563700 (LWP 6336)):\r\n#0  0x00007ffff761bd13 in epoll_wait () from /usr/lib64/libc.so.6\r\n#1  0x00007ffff6fe57e3 in epoll_dispatch (base=0x672060, tv=<optimized out>) at epoll.c:407\r\n#2  0x00007ffff6fe9230 in opal_libevent2022_event_base_loop (base=0x672060, flags=flags@entry=1) at event.c:1630\r\n#3  0x00007ffff6faa0de in progress_engine (obj=<optimized out>) at runtime/opal_progress_threads.c:105\r\n#4  0x00007ffff78ecdc5 in start_thread () from /usr/lib64/libpthread.so.0\r\n#5  0x00007ffff761b73d in clone () from /usr/lib64/libc.so.6\r\nThread 3 (Thread 0x7ffff2e86700 (LWP 6347)):\r\n#0  0x00007ffff761bd13 in epoll_wait () from /usr/lib64/libc.so.6\r\n#1  0x00007ffff6fe57e3 in epoll_dispatch (base=0x69dc20, tv=<optimized out>) at epoll.c:407\r\n#2  0x00007ffff6fe9230 in opal_libevent2022_event_base_loop (base=0x69dc20, flags=flags@entry=1) at event.c:1630\r\n#3  0x00007ffff350c21e in progress_engine (obj=<optimized out>) at runtime/pmix_progress_threads.c:109\r\n#4  0x00007ffff78ecdc5 in start_thread () from /usr/lib64/libpthread.so.0\r\n#5  0x00007ffff761b73d in clone () from /usr/lib64/libc.so.6\r\nThread 2 (Thread 0x7fffe3fff700 (LWP 6642)):\r\n#0  0x00007ffff761bd13 in epoll_wait () from /usr/lib64/libc.so.6\r\n#1  0x00007fffe8c73fb4 in ucs_async_thread_func (arg=0x80b9b0) at async/thread.c:93\r\n#2  0x00007ffff78ecdc5 in start_thread () from /usr/lib64/libpthread.so.0\r\n#3  0x00007ffff761b73d in clone () from /usr/lib64/libc.so.6\r\nThread 1 (Thread 0x7ffff7fb6740 (LWP 6306)):\r\n#0  0x00007fffeb53aacd in mlx5_poll_cq_1 () from /usr/lib64/libmlx5-rdmav2.so\r\n#1  0x00007fffe91bc7d3 in ibv_poll_cq (wc=0x7fffffffbb70, num_entries=<optimized out>, cq=<optimized out>) at /usr/include/infiniband/verbs.h:1458\r\n#2  uct_ib_poll_cq (wcs=0x7fffffffbb70, count=<synthetic pointer>, cq=<optimized out>) at /hpc/local/benchmarks/hpcx_install_Monday/src/hpcx-gcc-redhat7.2/ucx-master/src/uct/ib/base/ib_device.h:267\r\n#3  uct_rc_verbs_iface_poll_tx (iface=0x84c8f0) at ib/rc/verbs/rc_verbs_iface.c:110\r\n#4  uct_rc_verbs_iface_progress_tm (arg=0x84c8f0) at ib/rc/verbs/rc_verbs_iface.c:401\r\n#5  0x00007fffe940e7f2 in ucs_callbackq_dispatch (cbq=<optimized out>) at /hpc/local/benchmarks/hpcx_install_Monday/src/hpcx-gcc-redhat7.2/ucx-master/src/ucs/datastruct/callbackq.h:163\r\n#6  uct_worker_progress (worker=<optimized out>) at /hpc/local/benchmarks/hpcx_install_Monday/src/hpcx-gcc-redhat7.2/ucx-master/src/uct/api/uct.h:1412\r\n#7  ucp_worker_progress (worker=0x7fffe80c8010) at core/ucp_worker.c:1143\r\n#8  0x00007fffe962d237 in mca_pml_ucx_progress () at pml_ucx.c:451\r\n#9  0x00007ffff6fa45bc in opal_progress () at runtime/opal_progress.c:222\r\n#10 0x00007ffff7b48c75 in ompi_request_wait_completion (req=0xa78608) at ../ompi/request/request.h:392\r\n#11 ompi_request_default_wait (req_ptr=0x7fffffffc0a0, status=0x0) at request/req_wait.c:42\r\n#12 0x00007ffff7b6ca1a in PMPI_Wait (request=0x7fffffffc0a0, status=<optimized out>) at pwait.c:71\r\n#13 0x0000000000402613 in MTest_Allgather ()\r\n#14 0x000000000040293b in main ()\r\n\r\nnode=clx-hercules-069, pid=6305:\r\nThread 4 (Thread 0x7ffff4563700 (LWP 6333)):\r\n#0  0x00007ffff761bd13 in epoll_wait () from /usr/lib64/libc.so.6\r\n#1  0x00007ffff6fe57e3 in epoll_dispatch (base=0x672060, tv=<optimized out>) at epoll.c:407\r\n#2  0x00007ffff6fe9230 in opal_libevent2022_event_base_loop (base=0x672060, flags=flags@entry=1) at event.c:1630\r\n#3  0x00007ffff6faa0de in progress_engine (obj=<optimized out>) at runtime/opal_progress_threads.c:105\r\n#4  0x00007ffff78ecdc5 in start_thread () from /usr/lib64/libpthread.so.0\r\n#5  0x00007ffff761b73d in clone () from /usr/lib64/libc.so.6\r\nThread 3 (Thread 0x7ffff2e86700 (LWP 6346)):\r\n#0  0x00007ffff761bd13 in epoll_wait () from /usr/lib64/libc.so.6\r\n#1  0x00007ffff6fe57e3 in epoll_dispatch (base=0x69dc20, tv=<optimized out>) at epoll.c:407\r\n#2  0x00007ffff6fe9230 in opal_libevent2022_event_base_loop (base=0x69dc20, flags=flags@entry=1) at event.c:1630\r\n#3  0x00007ffff350c21e in progress_engine (obj=<optimized out>) at runtime/pmix_progress_threads.c:109\r\n#4  0x00007ffff78ecdc5 in start_thread () from /usr/lib64/libpthread.so.0\r\n#5  0x00007ffff761b73d in clone () from /usr/lib64/libc.so.6\r\nThread 2 (Thread 0x7fffe3fff700 (LWP 6635)):\r\n#0  0x00007ffff761bd13 in epoll_wait () from /usr/lib64/libc.so.6\r\n#1  0x00007fffe8c73fb4 in ucs_async_thread_func (arg=0x80b9b0) at async/thread.c:93\r\n#2  0x00007ffff78ecdc5 in start_thread () from /usr/lib64/libpthread.so.0\r\n#3  0x00007ffff761b73d in clone () from /usr/lib64/libc.so.6\r\nThread 1 (Thread 0x7ffff7fb6740 (LWP 6305)):\r\n#0  0x00007fffeb527fb0 in pthread_spin_unlock@plt () from /usr/lib64/libmlx5-rdmav2.so\r\n#1  0x00007fffeb53b21c in mlx5_poll_cq_1 () from /usr/lib64/libmlx5-rdmav2.so\r\n#2  0x00007fffe91bc7d3 in ibv_poll_cq (wc=0x7fffffffba60, num_entries=<optimized out>, cq=<optimized out>) at /usr/include/infiniband/verbs.h:1458\r\n#3  uct_ib_poll_cq (wcs=0x7fffffffba60, count=<synthetic pointer>, cq=<optimized out>) at /hpc/local/benchmarks/hpcx_install_Monday/src/hpcx-gcc-redhat7.2/ucx-master/src/uct/ib/base/ib_device.h:267\r\n#4  uct_rc_verbs_iface_poll_tx (iface=0x84c8f0) at ib/rc/verbs/rc_verbs_iface.c:110\r\n#5  uct_rc_verbs_iface_progress_tm (arg=0x84c8f0) at ib/rc/verbs/rc_verbs_iface.c:401\r\n#6  0x00007fffe940e7f2 in ucs_callbackq_dispatch (cbq=<optimized out>) at /hpc/local/benchmarks/hpcx_install_Monday/src/hpcx-gcc-redhat7.2/ucx-master/src/ucs/datastruct/callbackq.h:163\r\n#7  uct_worker_progress (worker=<optimized out>) at /hpc/local/benchmarks/hpcx_install_Monday/src/hpcx-gcc-redhat7.2/ucx-master/src/uct/api/uct.h:1412\r\n#8  ucp_worker_progress (worker=0x7fffe80c8010) at core/ucp_worker.c:1143\r\n#9  0x00007fffe962d237 in mca_pml_ucx_progress () at pml_ucx.c:451\r\n#10 0x00007ffff6fa45bc in opal_progress () at runtime/opal_progress.c:222\r\n#11 0x00007ffff7b488a3 in ompi_request_default_test_all (count=<optimized out>, requests=0xd79ed0, completed=<optimized out>, statuses=<optimized out>) at request/req_test.c:214\r\n#12 0x00007fffe85b61ba in NBC_Progress (handle=handle@entry=0xa78608) at nbc.c:329\r\n#13 0x00007fffe85b561f in ompi_coll_libnbc_progress () at coll_libnbc_component.c:275\r\n#14 0x00007ffff6fa45bc in opal_progress () at runtime/opal_progress.c:222\r\n#15 0x00007ffff7b48c75 in ompi_request_wait_completion (req=0xa78608) at ../ompi/request/request.h:392\r\n#16 ompi_request_default_wait (req_ptr=0x7fffffffc0a0, status=0x0) at request/req_wait.c:42\r\n#17 0x00007ffff7b6ca1a in PMPI_Wait (request=0x7fffffffc0a0, status=<optimized out>) at pwait.c:71\r\n#18 0x0000000000402613 in MTest_Allgather ()\r\n#19 0x000000000040293b in main ()\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/openucx/ucx/issues/comments/333373734","html_url":"https://github.com/openucx/ucx/issues/1760#issuecomment-333373734","issue_url":"https://api.github.com/repos/openucx/ucx/issues/1760","id":333373734,"user":{"login":"yosefe","id":2255631,"avatar_url":"https://avatars0.githubusercontent.com/u/2255631?v=4","gravatar_id":"","url":"https://api.github.com/users/yosefe","html_url":"https://github.com/yosefe","followers_url":"https://api.github.com/users/yosefe/followers","following_url":"https://api.github.com/users/yosefe/following{/other_user}","gists_url":"https://api.github.com/users/yosefe/gists{/gist_id}","starred_url":"https://api.github.com/users/yosefe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yosefe/subscriptions","organizations_url":"https://api.github.com/users/yosefe/orgs","repos_url":"https://api.github.com/users/yosefe/repos","events_url":"https://api.github.com/users/yosefe/events{/privacy}","received_events_url":"https://api.github.com/users/yosefe/received_events","type":"User","site_admin":false},"created_at":"2017-10-01T12:36:12Z","updated_at":"2017-10-01T12:36:12Z","author_association":"CONTRIBUTOR","body":"fixed in new driver version"}}
{"actor":{"display_login":"yosefe"}}