title: Why is CMake apparently referring to host system files while cross-compiling a Net-SNMP agent despite a proper toolchain file is being used?
body: <p>I am developing a <a href="http://www.net-snmp.org/" rel="nofollow noreferrer">Net-SNMP</a> subagent whose final target will be an ARM board, so I'm using <a href="https://cmake.org/" rel="nofollow noreferrer">CMake</a> to make it easier to manage building the native and cross-compiled versions. I started testing it on my host platform (x86_64) and it builds and works fine with this <strong>CMakeLists.txt</strong> file:</p>
  
  
  
  <pre class="lang-bash prettyprint-override"><code>cmake_minimum_required (VERSION 2.6)
  
  project (snmp_agent C)
  
  set(snmp_agent_VERSION_MAJOR 1)
  set(snmp_agent_VERSION_MINOR 0)
  
  # Defines path to the net-snmp-config script
  set(NETSNMPCONFIG "${CMAKE_FIND_ROOT_PATH}/usr/bin/net-snmp-config")
  
  # Gets compiling flags and libs linked to Net-SNMP
  execute_process(COMMAND "${NETSNMPCONFIG}" "--base-cflags" OUTPUT_VARIABLE NETSNMPCFLAGS)
  execute_process(COMMAND "${NETSNMPCONFIG}" "--agent-libs" OUTPUT_VARIABLE NETSNMPLIBS)
  
  # Removes leading/trailing spaces from net-snmp-config output
  string(STRIP ${NETSNMPCFLAGS} NETSNMPCFLAGS)
  string(STRIP ${NETSNMPLIBS} NETSNMPLIBS)
  
  # Prints compilation and linker flags used in Net-SNMP package
  message("Net-SNMP package CFLAGS: ${NETSNMPCFLAGS}")
  message("Net-SNMP package LIBS: ${NETSNMPLIBS}")
  
  # Setting libs and compilation flags variables
  set(LIBS "${NETSNMPLIBS}")
  set(STRICT_FLAGS "-Wall -Wstrict-prototypes")
  set(CFLAGS "-I. ${STRICT_FLAGS} ${NETSNMPCFLAGS}")
  
  # Sets prefix for files created by 'mib2c' for the wanted MIB
  set(ENT_PHYSICAL_ENTRY "scalars/entPhysicalEntry")
  
  # Source files created by 'mib2c' and then user customized
  set(USER_SRCS
      ${ENT_PHYSICAL_ENTRY}.c
    )
  
  # Setting subagent sources
  set(SRCS ${USER_SRCS}
      ${CMAKE_PROJECT_NAME}.c
    )
  
  # Finds the required Net-SNMP lib paths and assigns them to variables
  find_library(NETSNMPAGENT "netsnmpagent")
  message("Found ${NETSNMPAGENT}")
  find_library(NETSNMPMIBS "netsnmpmibs")
  message("Found ${NETSNMPMIBS}")
  find_library(NETSNMP "netsnmp")
  message("Found ${NETSNMP}")
  
  # Sets the flags to be used for compiling and linking the executable
  set_source_files_properties(${SRCS} COMPILE_FLAGS ${CFLAGS})
  add_executable(${CMAKE_PROJECT_NAME} ${SRCS})
  target_link_libraries(${CMAKE_PROJECT_NAME} ${NETSNMPAGENT} ${NETSNMPMIBS} ${NETSNMP})
  </code></pre>
  
  <p>I use a separate build dir so I don't mix build and source files, which stay along with the <strong>CMakeLists.txt</strong> file, so the the output of these commands...</p>
  
  <pre class="lang-bash prettyprint-override"><code>cd ~/git/snmp_agent # CMakeLists.txt is in here along with source files
  mkdir build
  cd build
  cmake ..
  </code></pre>
  
  <p>...is as follows:</p>
  
  <pre class="lang-bash prettyprint-override"><code>claudio@slackdev:~/git/snmp_agent/build$ cmake ..
  -- The C compiler identification is GNU 5.3.0
  -- Check for working C compiler: /usr/bin/cc
  -- Check for working C compiler: /usr/bin/cc -- works
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Detecting C compile features
  -- Detecting C compile features - done
  Net-SNMP package CFLAGS: -DNETSNMP_ENABLE_IPV6 -fno-strict-aliasing -O2 -fPIC -Ulinux -Dlinux=linux -D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -I/usr/lib64/perl5/CORE -I/usr/include/libnl3 -I/usr/include
  Net-SNMP package LIBS: -L/usr/lib64 -lnetsnmpmibs -lsensors -lpci -ldl -lnetsnmpagent -lwrap -lnsl -Wl,-E -Wl,-rpath,/usr/lib64/perl5/CORE -lnetsnmp -lcrypto -lnl-3 -lm
  Found /usr/lib64/libnetsnmpagent.so
  Found /usr/lib64/libnetsnmpmibs.so
  Found /usr/lib64/libnetsnmp.so
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /home/claudio/git/snmp_agent/build
  </code></pre>
  
  <p>Running <code>make</code>, it builds just fine:</p>
  
  <pre class="lang-bash prettyprint-override"><code>claudio@slackdev:~/git/snmp_agent/build$ make
  Scanning dependencies of target snmp_agent
  [ 25%] Building C object CMakeFiles/snmp_agent.dir/scalars/entPhysicalEntry.c.o
  [ 50%] Building C object CMakeFiles/snmp_agent.dir/snmp_agent.c.o
  [100%] Linking C executable snmp_agent
  [100%] Built target snmp_agent
  </code></pre>
  
  <p>This <code>snmp_agent</code> runs as an agentX subagent, connecting to the main <code>snmpd</code> master agent and properly answering SNMP requests against <code>entPhysicalEntry</code> scalar objects.</p>
  
  <p>Ok, so far so good. The problem is when I try to cross-compile this so it does the same thing on my final target, which is an ARMv7 Allwinner A20 board. I already have a cross-toolchain available for it - which I created using <a href="http://crosstool-ng.github.io/" rel="nofollow noreferrer">Crosstool-NG</a>. It is in the search PATH and its tools begin with the prefix <code>armv7-a20_allwinner-linux-gnueabihf</code>. In order to use it, I set up the following <a href="https://cmake.org/Wiki/CMake_Cross_Compiling#The_toolchain_file" rel="nofollow noreferrer">toolchain file</a>, which was named <code>armv7-a20_allwinner-linux-gnueabihf.cmake</code>:</p>
  
  <pre class="lang-bash prettyprint-override"><code># the name of the target operating system
  SET(CMAKE_SYSTEM_NAME Linux)
  
  # which C and C++ compiler to use
  SET(CMAKE_C_COMPILER   armv7-a20_allwinner-linux-gnueabihf-gcc)
  SET(CMAKE_CXX_COMPILER armv7-a20_allwinner-linux-gnueabihf-g++) 
  
  # here is the target environment located
  SET(CMAKE_FIND_ROOT_PATH  "$ENV{HOME}/arm_rootfs") 
  
  # adjust the default behaviour of the FIND_XXX() commands:
  # search headers and libraries in the target environment, search 
  # programs in the host environment
  set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
  set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
  set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
  </code></pre>
  
  <p>Using a different build dir for the ARM version, CMake should pick up the right paths pointing to the ARM root filesystem, which is a copy of what will be on the target board's SDcard:</p>
  
  <pre class="lang-bash prettyprint-override"><code>cd ~/git/snmp_agent # CMakeLists.txt and armv7-a20_allwinner-linux-gnueabihf.cmake are here
  mkdir build-arm
  cd build-arm
  cmake -DCMAKE_TOOLCHAIN_FILE=../armv7-a20_allwinner-linux-gnueabihf.cmake ..
  </code></pre>
  
  <p>Note that apparently CMake runs fine, as the paths found are all relative to the ARM root filesystem:</p>
  
  <pre class="lang-bash prettyprint-override"><code>claudio@slackdev:~/git/snmp_agent/build-arm$ cmake -DCMAKE_TOOLCHAIN_FILE=../cmake_defs/armv7-a20_allwinner-linux-gnueabihf.cmake ..
  -- The C compiler identification is GNU 6.3.0
  -- Check for working C compiler: /home/claudio/x-tools/armv7-a20_allwinner-linux-gnueabihf/bin/armv7-a20_allwinner-linux-gnueabihf-gcc
  -- Check for working C compiler: /home/claudio/x-tools/armv7-a20_allwinner-linux-gnueabihf/bin/armv7-a20_allwinner-linux-gnueabihf-gcc -- works
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Detecting C compile features
  -- Detecting C compile features - done
  Net-SNMP package CFLAGS: -DNETSNMP_ENABLE_IPV6 -fno-strict-aliasing -O2 -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard -Ulinux -Dlinux=linux -D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -I/usr/lib/perl5/CORE -I/usr/include/libnl3 -I/usr/include
  Net-SNMP package LIBS: -L/usr/lib -lnetsnmpmibs -lsensors -lpci -ldl -lnetsnmpagent -lwrap -lnsl -Wl,-E -Wl,-rpath,/usr/lib/perl5/CORE -lnetsnmp -lcrypto -lnl-3 -lm
  Found /home/claudio/arm_rootfs/usr/lib/libnetsnmpagent.so
  Found /home/claudio/arm_rootfs/usr/lib/libnetsnmpmibs.so
  Found /home/claudio/arm_rootfs/usr/lib/libnetsnmp.so
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /home/claudio/git/snmp_agent/build-arm
  </code></pre>
  
  <p>But when running <code>make</code> it falls apart while looking for <code>/usr/include/gnu/stubs.h</code>. If we try to repeat <code>make</code> with <code>VERBOSE=1</code>, we can see the gcc invocation is using paths relative to the host machine:</p>
  
  <pre class="lang-bash prettyprint-override"><code>claudio@slackdev:~/git/snmp_agent/build-arm$ make VERBOSE=1
  /usr/bin/cmake -H/home/claudio/git/snmp_agent -B/home/claudio/git/snmp_agent/build-arm --check-build-system CMakeFiles/Makefile.cmake 0
  /usr/bin/cmake -E cmake_progress_start /home/claudio/git/snmp_agent/build-arm/CMakeFiles /home/claudio/git/snmp_agent/build-arm/CMakeFiles/progress.marks
  make -f CMakeFiles/Makefile2 all
  make[1]: Entering directory 'snmp_agent/build-arm'
  make -f CMakeFiles/snmp_agent.dir/build.make CMakeFiles/snmp_agent.dir/depend
  make[2]: Entering directory 'snmp_agent/build-arm'
  cd /home/claudio/git/snmp_agent/build-arm &amp;&amp; /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/claudio/git/snmp_agent /home/claudio/git/snmp_agent /home/claudio/git/snmp_agent/build-arm /home/claudio/git/snmp_agent/build-arm /home/claudio/git/snmp_agent/build-arm/CMakeFiles/snmp_agent.dir/DependInfo.cmake --color=
  make[2]: Leaving directory 'snmp_agent/build-arm'
  make -f CMakeFiles/snmp_agent.dir/build.make CMakeFiles/snmp_agent.dir/build
  make[2]: Entering directory 'snmp_agent/build-arm'
  [ 25%] Building C object CMakeFiles/snmp_agent.dir/scalars/entPhysicalEntry.c.o
  /home/claudio/x-tools/armv7-a20_allwinner-linux-gnueabihf/bin/armv7-a20_allwinner-linux-gnueabihf-gcc     -I. -Wall -Wstrict-prototypes -DNETSNMP_ENABLE_IPV6 -fno-strict-aliasing -O2 -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard -Ulinux -Dlinux=linux -D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -I/usr/lib/perl5/CORE -I/usr/include/libnl3 -I/usr/include -o CMakeFiles/snmp_agent.dir/scalars/entPhysicalEntry.c.o   -c /home/claudio/git/snmp_agent/scalars/entPhysicalEntry.c
  In file included from /usr/include/features.h:392:0,
                   from /usr/include/stdio.h:27,
                   from /usr/include/net-snmp/net-snmp-includes.h:14,
                   from /home/claudio/git/snmp_agent/scalars/entPhysicalEntry.c:7:
  /usr/include/gnu/stubs.h:7:27: fatal error: gnu/stubs-32.h: No such file or directory
   # include &lt;gnu/stubs-32.h&gt;
                             ^
  compilation terminated.
  CMakeFiles/snmp_agent.dir/build.make:62: recipe for target 'CMakeFiles/snmp_agent.dir/scalars/entPhysicalEntry.c.o' failed
  make[2]: *** [CMakeFiles/snmp_agent.dir/scalars/entPhysicalEntry.c.o] Error 1
  make[2]: Leaving directory 'snmp_agent/build-arm'
  CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/snmp_agent.dir/all' failed
  make[1]: *** [CMakeFiles/snmp_agent.dir/all] Error 2
  make[1]: Leaving directory 'snmp_agent/build-arm'
  Makefile:83: recipe for target 'all' failed
  make: *** [all] Error 2
  </code></pre>
  
  <p>The strange thing here is the <code>/usr/include/gnu/stubs.h</code> from the ARM root filesystem has nothing to do with <code>stubs-32.h</code>:</p>
  
  <pre class="lang-c prettyprint-override"><code>/* This file is automatically generated.
     This file selects the right generated file of `__stub_FUNCTION' macros
     based on the architecture being compiled for.  */
  
  
  #if !defined __ARM_PCS_VFP
  # include &lt;gnu/stubs-soft.h&gt;
  #endif
  #if defined __ARM_PCS_VFP
  # include &lt;gnu/stubs-hard.h&gt;
  #endif
  </code></pre>
  
  <p>But if you look at the same file from the host system - which is a x86_64 machine - we can guess why it tried to find <code>stubs-32.h</code> (remember we are trying to compile for ARM, so it won't find the <code>__x86_64__</code> symbol definition):</p>
  
  <pre class="lang-c prettyprint-override"><code>/* This file is automatically generated.
     This file selects the right generated file of `__stub_FUNCTION' macros
     based on the architecture being compiled for.  */
  
  
  #if !defined __x86_64__
  # include &lt;gnu/stubs-32.h&gt;
  #endif
  #if defined __x86_64__ &amp;&amp; defined __LP64__
  # include &lt;gnu/stubs-64.h&gt;
  #endif
  #if defined __x86_64__ &amp;&amp; defined __ILP32__
  # include &lt;gnu/stubs-x32.h&gt;
  #endif
  </code></pre>
  
  <p>Why is that happening, since the toolchain file clearly specifies that libraries and includes are to be searched ONLY in the path set by CMAKE_FIND_ROOT_PATH?</p>
  
  <p><strong>UPDATE (problem not totally solved yet!):</strong></p>
  
  <p>After I had accepted the answer from @Tsyvarev I was double checking my <strong>CMakeLists.txt</strong> file and I found while trying to make it work I had made a hack by manually setting CMAKE_FIND_ROOT_PATH as prefix of each compiler include switches (-I) returned by <strong>net-snmp-config</strong>, which obviously is not the ideal thing to do. That, along with CMAKE_SYSROOT, makes it work, but CMAKE_SYSROOT alone doesn't prefix the include paths by itself: </p>
  
  <pre class="lang-bash prettyprint-override"><code># This is the manually hacked line:
  set(NETSNMPCFLAGS "-DNETSNMP_ENABLE_IPV6 -fno-strict-aliasing -O2 -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard -Ulinux -Dlinux=linux -D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I${CMAKE_FIND_ROOT_PATH}/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2 -I${CMAKE_FIND_ROOT_PATH}/usr/lib/perl5/CORE -I${CMAKE_FIND_ROOT_PATH}/usr/include/libnl3 -I${CMAKE_FIND_ROOT_PATH}/usr/include")
  
  set(LIBS "${NETSNMPLIBS}")
  
  set(STRICT_FLAGS "-Wall -Wstrict-prototypes")
  set(CFLAGS "-I. ${STRICT_FLAGS} ${NETSNMPCFLAGS}")
  </code></pre>