{"action":"created","issue":{"url":"https://api.github.com/repos/STEllAR-GROUP/hpx/issues/2837","repository_url":"https://api.github.com/repos/STEllAR-GROUP/hpx","labels_url":"https://api.github.com/repos/STEllAR-GROUP/hpx/issues/2837/labels{/name}","comments_url":"https://api.github.com/repos/STEllAR-GROUP/hpx/issues/2837/comments","events_url":"https://api.github.com/repos/STEllAR-GROUP/hpx/issues/2837/events","html_url":"https://github.com/STEllAR-GROUP/hpx/issues/2837","id":250583242,"number":2837,"title":"Unresolved extern variables and Segmentation fault with CUDA Clang++","user":{"login":"Element-126","id":8004066,"avatar_url":"https://avatars1.githubusercontent.com/u/8004066?v=4","gravatar_id":"","url":"https://api.github.com/users/Element-126","html_url":"https://github.com/Element-126","followers_url":"https://api.github.com/users/Element-126/followers","following_url":"https://api.github.com/users/Element-126/following{/other_user}","gists_url":"https://api.github.com/users/Element-126/gists{/gist_id}","starred_url":"https://api.github.com/users/Element-126/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Element-126/subscriptions","organizations_url":"https://api.github.com/users/Element-126/orgs","repos_url":"https://api.github.com/users/Element-126/repos","events_url":"https://api.github.com/users/Element-126/events{/privacy}","received_events_url":"https://api.github.com/users/Element-126/received_events","type":"User","site_admin":false},"labels":[{"id":372083494,"url":"https://api.github.com/repos/STEllAR-GROUP/hpx/labels/platform:%20CUDA","name":"platform: CUDA","color":"006b75","default":false},{"id":12168885,"url":"https://api.github.com/repos/STEllAR-GROUP/hpx/labels/type:%20defect","name":"type: defect","color":"e10c02","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/STEllAR-GROUP/hpx/milestones/26","html_url":"https://github.com/STEllAR-GROUP/hpx/milestone/26","labels_url":"https://api.github.com/repos/STEllAR-GROUP/hpx/milestones/26/labels","id":2254928,"number":26,"title":"1.1.0","description":"","creator":{"login":"hkaiser","id":568063,"avatar_url":"https://avatars1.githubusercontent.com/u/568063?v=4","gravatar_id":"","url":"https://api.github.com/users/hkaiser","html_url":"https://github.com/hkaiser","followers_url":"https://api.github.com/users/hkaiser/followers","following_url":"https://api.github.com/users/hkaiser/following{/other_user}","gists_url":"https://api.github.com/users/hkaiser/gists{/gist_id}","starred_url":"https://api.github.com/users/hkaiser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hkaiser/subscriptions","organizations_url":"https://api.github.com/users/hkaiser/orgs","repos_url":"https://api.github.com/users/hkaiser/repos","events_url":"https://api.github.com/users/hkaiser/events{/privacy}","received_events_url":"https://api.github.com/users/hkaiser/received_events","type":"User","site_admin":false},"open_issues":134,"closed_issues":221,"state":"open","created_at":"2017-01-16T14:53:21Z","updated_at":"2017-08-16T16:20:13Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2017-08-16T10:41:25Z","updated_at":"2017-08-16T16:26:14Z","closed_at":null,"body":"Given the current build issues with NVCC 8.0 & 9.0, I decided to try my luck and use Clang++ to compile both host and device code.\r\n\r\nI configured HPX with the following parameters:\r\n```cmake\r\ncmake -D CMAKE_INSTALL_PREFIX=/home/jtastet/.local/stow/hpx_cuda-clang \\\r\n      -D CMAKE_CXX_COMPILER=clang++ \\\r\n      -D CMAKE_C_COMPILER=clang \\\r\n      -D HPX_WITH_CUDA=ON \\\r\n      -D HPX_WITH_CUDA_CLANG=ON \\\r\n      -D HPX_WITH_MALLOC=jemalloc \\\r\n      -D HPX_WITH_PARCELPORT_MPI=ON \\\r\n      -D HPX_WITH_CUDA_ARCH=\"sm_35;sm_61\" \\\r\n..\r\n```\r\n\r\nBoth HPX and all of its dependencies were compiled using Clang trunk (6.0.0, rev. `310909`).\r\n\r\nHPX compiled fine, except for the following warnings related to `ptxas`:\r\n\r\n```\r\nScanning dependencies of target benchmark_unique_copy_exe\r\n[ 85%] Building CXX object tests/performance/parallel_algorithms/local/CMakeFiles/benchmark_unique_copy_exe.dir/benchmark_unique_copy.cpp.o\r\nptxas warning : Unresolved extern variable '_ZTVN10__cxxabiv120__si_class_type_infoE' in whole program compilation, ignoring extern qualifier\r\nptxas warning : Unresolved extern variable '_ZTVN10__cxxabiv117__class_type_infoE' in whole program compilation, ignoring extern qualifier\r\n...\r\nScanning dependencies of target 1d_stencil_3_exe\r\n[ 89%] Building CXX object examples/1d_stencil/CMakeFiles/1d_stencil_3_exe.dir/1d_stencil_3.cpp.o\r\nptxas warning : Unresolved extern variable '_ZTVN10__cxxabiv117__class_type_infoE' in whole program compilation, ignoring extern qualifier\r\nptxas warning : Unresolved extern variable '_ZTVN10__cxxabiv120__si_class_type_infoE' in whole program compilation, ignoring extern qualifier\r\n```\r\nFor reference, the full build log is [here](https://github.com/STEllAR-GROUP/hpx/files/1227774/hpx_cuda_clang_build_log.txt).\r\n\r\nI am well aware that the test suite currently fails to compile and run due to issues #2217 and #2690. But here the problem seems to be something else.\r\n\r\nTo test the resulting HPX build, I wrote a trivial test program:\r\n```cuda\r\n#include <hpx/hpx.hpp>\r\n#include <hpx/hpx_init.hpp>\r\n\r\n#include <cuda.h>\r\n\r\n#include <vector>\r\n#include <cassert>\r\n\r\n__global__ void increment_kernel(int* x)\r\n{\r\n    *x += 1;\r\n}\r\n\r\n__host__ int increment(int x_h)\r\n{\r\n    int* x_d = nullptr;\r\n    auto err1 = cudaMalloc(&x_d, sizeof(int));\r\n    assert(err1 == cudaSuccess);\r\n    auto err2 = cudaMemcpy(x_d, &x_h, sizeof(int), cudaMemcpyHostToDevice);\r\n    assert(err2 == cudaSuccess);\r\n    increment_kernel<<<1, 1>>>(x_d);\r\n    auto err3 = cudaMemcpy(&x_h, x_d, sizeof(int), cudaMemcpyDeviceToHost);\r\n    assert(err3 == cudaSuccess);\r\n    return x_h;\r\n}\r\n\r\nHPX_PLAIN_ACTION(increment, increment_action);\r\n\r\n__host__ int hpx_main(int argc, char* argv[])\r\n{\r\n    auto fut = hpx::async(increment_action(), hpx::find_here(), 0);\r\n\r\n    assert(fut.get() == 1);\r\n\r\n    return hpx::finalize();\r\n}\r\n\r\n__host__ int main(int argc, char* argv[])\r\n{\r\n    return hpx::init(argc, argv);\r\n}\r\n```\r\n\r\nClang emits the same ptxas-related warnings (+ a new one) when compiling it:\r\n```\r\n$ clang++ -o test test.cu --cuda-path=/usr/local/cuda-8.0 `pkg-config --cflags --libs hpx_application` -lhpx_iostreams -D HPX_APPLICATION\r\nptxas warning : Unresolved extern variable '_ZN3hpx7actions6detail32register_action_invocation_countI16increment_actionE8instanceE' in whole program compilation, ignoring extern qualifier\r\nptxas warning : Unresolved extern variable '_ZTVN10__cxxabiv117__class_type_infoE' in whole program compilation, ignoring extern qualifier\r\nptxas warning : Unresolved extern variable '_ZTVN10__cxxabiv120__si_class_type_infoE' in whole program compilation, ignoring extern qualifier\r\n```\r\n\r\nand executing it leads to a segmentation fault:\r\n```\r\n$ ./test\r\n{stack-trace}: 16 frames:\r\n0x7f455b909876  : hpx::util::stack_trace::trace(void**, unsigned long) + 0x6 in /home/jtastet/.local/lib/libhpx.so.1\r\n0x7f455b1d7ecb  : hpx::util::backtrace::backtrace(unsigned long) + 0x5b in /home/jtastet/.local/lib/libhpx.so.1\r\n0x7f455b21f8ee  : hpx::termination_handler(int) + 0x19e in /home/jtastet/.local/lib/libhpx.so.1\r\n0x7f45595d6370  : ??? + 0x7f45595d6370 in /lib64/libpthread.so.0\r\n0x484877        : ??? + 0x484877 in ./test\r\n0x484255        : hpx::components::managed_component<hpx::lcos::detail::promise_lco<int, int>, hpx::components::detail::this_type>::operator new(unsigned long) + 0x35 in ./test\r\n0x480e89        : hpx::lcos::detail::promise_base<int, int, hpx::lcos::detail::promise_data<int> >::init_shared_state() + 0x29 in ./test\r\n0x480d85        : hpx::lcos::detail::promise_base<int, int, hpx::lcos::detail::promise_data<int> >::promise_base() + 0x55 in ./test\r\n0x7f455b6837b0  : hpx::lcos::future<hpx::traits::extract_action<hpx::components::server::runtime_support::load_components_action, void>::local_result_type> hpx::detail::async_impl<hpx::components::s\r\nerver::runtime_support::load_components_action>(hpx::detail::async_policy, hpx::naming::id_type const&) + 0x60 in /home/jtastet/.local/lib/libhpx.so.1\r\n0x7f455b6710ee  : hpx::components::stubs::runtime_support::load_components(hpx::naming::id_type const&) + 0x1e in /home/jtastet/.local/lib/libhpx.so.1\r\n0x7f455b22b17f  : hpx::pre_main(hpx::runtime_mode) + 0x1ef in /home/jtastet/.local/lib/libhpx.so.1\r\n0x7f455b161768  : hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::pol\r\nicies::lockfree_lifo> >::run_helper(hpx::util::function<int (), false>, int&) + 0x68 in /home/jtastet/.local/lib/libhpx.so.1\r\n0x7f455b1b0c0c  : _ZN3hpx4util6detail10bound_implIMNS_12runtime_implINS_7threads8policies30local_priority_queue_schedulerISt5mutexNS5_13lockfree_fifoES8_NS5_13lockfree_lifoEEEEEFSt4pairINS4_17thread\r\n_state_enumEN5boost13intrusive_ptrINS4_11thread_dataEEEENS0_8functionIFivELb0EEERiENS0_5tupleIJPSB_SL_SM_EEENSP_IJONS4_20thread_state_ex_enumEEEEJLm0ELm1ELm2EEEENSt9enable_ifIXsr6detail14is_simple_b\r\nindIT0_EE5valueENS1_19invoke_bound_resultIT_SW_T1_E4typeEE4typeERSY_RSW_OSZ_NS1_6pack_cImJXspT2_EEEE + 0x6c in /home/jtastet/.local/lib/libhpx.so.1\r\n0x7f455b1b0b8e  : std::pair<hpx::threads::thread_state_enum, boost::intrusive_ptr<hpx::threads::thread_data> > hpx::util::detail::callable_vtable<std::pair<hpx::threads::thread_state_enum, boost::in\r\ntrusive_ptr<hpx::threads::thread_data> > (hpx::threads::thread_state_ex_enum)>::_invoke<hpx::util::detail::bound<std::pair<hpx::threads::thread_state_enum, boost::intrusive_ptr<hpx::threads::thread_\r\ndata> > (hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lo\r\nckfree_lifo> >::*(hpx::runtime_impl<hpx::threads::policies::local_priority_queue_scheduler<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::pol\r\nicies::lockfree_lifo> >*&&, hpx::util::function<int (), false> const&, std::reference_wrapper<int>&&))(hpx::util::function<int (), false>, int&)> >(void**, hpx::threads::thread_state_ex_enum&&) + 0x\r\n1e in /home/jtastet/.local/lib/libhpx.so.1\r\n0x7f455b713132  : hpx::threads::coroutines::detail::coroutine_impl::operator()() + 0xc2 in /home/jtastet/.local/lib/libhpx.so.1\r\n0x7f455b19d9d6  : void hpx::threads::coroutines::detail::lx::trampoline<hpx::threads::coroutines::detail::coroutine_impl>(hpx::threads::coroutines::detail::coroutine_impl*) + 0x6 in /home/jtastet/.l\r\nocal/lib/libhpx.so.1\r\n{what}: Segmentation fault\r\n{config}:\r\n  HPX_WITH_AGAS_DUMP_REFCNT_ENTRIES=OFF\r\n  HPX_WITH_APEX=OFF\r\n  HPX_WITH_AUTOMATIC_SERIALIZATION_REGISTRATION=ON\r\n  HPX_WITH_AWAIT=OFF\r\n  HPX_WITH_CUDA=ON\r\n  HPX_WITH_CXX14_RETURN_TYPE_DEDUCTION=TRUE\r\n  HPX_WITH_DEPRECATION_WARNINGS=ON\r\n  HPX_WITH_EXECUTION_POLICY_COMPATIBILITY=ON\r\n  HPX_WITH_EXECUTOR_COMPATIBILITY=ON\r\n  HPX_WITH_GOOGLE_PERFTOOLS=OFF\r\n  HPX_WITH_HWLOC=ON\r\n  HPX_WITH_INCLUSIVE_SCAN_COMPATIBILITY=ON\r\n  HPX_WITH_IO_COUNTERS=ON\r\n  HPX_WITH_ITTNOTIFY=OFF\r\n  HPX_WITH_LOCAL_DATAFLOW_COMPATIBILITY=ON\r\n  HPX_WITH_LOGGING=ON\r\n  HPX_WITH_MORE_THAN_64_THREADS=(\r\n  HPX_WITH_NATIVE_TLS=ON\r\n  HPX_WITH_NETWORKING=ON\r\n  HPX_WITH_PAPI=OFF\r\n  HPX_WITH_PARCELPORT_ACTION_COUNTERS=OFF\r\n  HPX_WITH_PARCELPORT_LIBFABRIC=OFF\r\n  HPX_WITH_PARCELPORT_MPI=ON\r\n  HPX_WITH_PARCELPORT_MPI_MULTITHREADED=ON\r\n  HPX_WITH_PARCELPORT_TCP=ON\r\n  HPX_WITH_PARCELPORT_VERBS=OFF\r\n  HPX_WITH_PARCEL_COALESCING=ON\r\n  HPX_WITH_PARCEL_PROFILING=OFF\r\n  HPX_WITH_SCHEDULER_LOCAL_STORAGE=OFF\r\n  HPX_WITH_SPINLOCK_DEADLOCK_DETECTION=OFF\r\n  HPX_WITH_STACKTRACES=ON\r\n  HPX_WITH_SWAP_CONTEXT_EMULATION=OFF\r\n  HPX_WITH_THREAD_BACKTRACE_ON_SUSPENSION=OFF\r\n  HPX_WITH_THREAD_CREATION_AND_CLEANUP_RATES=OFF\r\n  HPX_WITH_THREAD_CUMULATIVE_COUNTS=ON\r\n  HPX_WITH_THREAD_DEBUG_INFO=OFF\r\n  HPX_WITH_THREAD_DESCRIPTION_FULL=OFF\r\n  HPX_WITH_THREAD_GUARD_PAGE=ON\r\n  HPX_WITH_THREAD_IDLE_RATES=OFF\r\n  HPX_WITH_THREAD_LOCAL_STORAGE=OFF\r\n  HPX_WITH_THREAD_MANAGER_IDLE_BACKOFF=ON\r\n  HPX_WITH_THREAD_QUEUE_WAITTIME=OFF\r\n  HPX_WITH_THREAD_STACK_MMAP=ON\r\n  HPX_WITH_THREAD_STEALING_COUNTS=ON\r\n  HPX_WITH_THREAD_TARGET_ADDRESS=OFF\r\n  HPX_WITH_TRANSFORM_REDUCE_COMPATIBILITY=ON\r\n  HPX_WITH_TUPLE_RVALUE_SWAP=ON\r\n  HPX_WITH_UNWRAPPED_COMPATIBILITY=ON\r\n  HPX_WITH_VALGRIND=OFF\r\n  HPX_WITH_VERIFY_LOCKS=OFF\r\n  HPX_WITH_VERIFY_LOCKS_BACKTRACE=OFF\r\n  HPX_WITH_VERIFY_LOCKS_GLOBALLY=OFF\r\n\r\n  HPX_PARCEL_MAX_CONNECTIONS=512\r\n  HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4\r\n  HPX_AGAS_LOCAL_CACHE_SIZE=4096\r\n  HPX_HAVE_MALLOC=jemalloc\r\n  HPX_PREFIX (configured)=/home/jtastet/.local/stow/hpx_cuda-clang\r\n  HPX_PREFIX=/home/jtastet/.local\r\n{version}: V1.1.0 (AGAS: V3.0), Git: d55dc1b220\r\n{boost}: V1.64.0\r\n{build-type}: release\r\n{date}: Aug 16 2017 10:54:00\r\n{platform}: linux\r\n{compiler}: Clang version 6.0.0 (trunk 310909)\r\n{stdlib}: GNU libstdc++ version 20150623\r\nAborted (core dumped)\r\n```\r\nI do not know for certain if the segmentation fault is related to the warnings issued by Clang, but it seems to be the most likely culprit.\r\n\r\nI am willing to help regarding all the CUDA-related issues, especially the ones which I am reporting, since I can easily reproduce them and investigate them further. However my knowledge of the HPX internals is still rather limited, so I might be missing some obvious aspects. So if you have any idea about what might be causing the issue, I would also greatly appreciate it.\r\n\r\nRegarding the present issue, I will try to find a minimal example (without HPX) which produces similar warnings as well as a segmentation fault."},"comment":{"url":"https://api.github.com/repos/STEllAR-GROUP/hpx/issues/comments/322826121","html_url":"https://github.com/STEllAR-GROUP/hpx/issues/2837#issuecomment-322826121","issue_url":"https://api.github.com/repos/STEllAR-GROUP/hpx/issues/2837","id":322826121,"user":{"login":"hkaiser","id":568063,"avatar_url":"https://avatars1.githubusercontent.com/u/568063?v=4","gravatar_id":"","url":"https://api.github.com/users/hkaiser","html_url":"https://github.com/hkaiser","followers_url":"https://api.github.com/users/hkaiser/followers","following_url":"https://api.github.com/users/hkaiser/following{/other_user}","gists_url":"https://api.github.com/users/hkaiser/gists{/gist_id}","starred_url":"https://api.github.com/users/hkaiser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hkaiser/subscriptions","organizations_url":"https://api.github.com/users/hkaiser/orgs","repos_url":"https://api.github.com/users/hkaiser/repos","events_url":"https://api.github.com/users/hkaiser/events{/privacy}","received_events_url":"https://api.github.com/users/hkaiser/received_events","type":"User","site_admin":false},"created_at":"2017-08-16T16:26:14Z","updated_at":"2017-08-16T16:26:14Z","body":"@Element-126 I agree, we don't have a reason to support C++11 mode if C++14 mode works."}}
{"actor":{"display_login":"hkaiser"}}