title: Tensorflow 2 Object Detection API: Numpy Version Errors
body: <p>I followed the &quot;Training Custom Object Detector&quot; tutorial (<a href="https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html" rel="noreferrer">https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html</a>)</p>
  <p>When running the script to continue training a pre-trained model:</p>
  <p><code>python model_main_tf2.py --model_dir=models/my_ssd_resnet50_v1_fpn --pipeline_config_path=models/my_ssd_resnet50_v1_fpn/pipeline.config</code></p>
  <p>(found here: <a href="https://github.com/tensorflow/models/blob/master/research/object_detection/model_main_tf2.py" rel="noreferrer">https://github.com/tensorflow/models/blob/master/research/object_detection/model_main_tf2.py</a>)</p>
  <p><strong>Using different numpy versions</strong>, I get the following errors.</p>
  <p>Scenario #1:</p>
  <ul>
  <li>tensorflow: 2.2.0</li>
  <li>numpy: 1.20.0-1</li>
  </ul>
  <p><code>NotImplementedError: Cannot convert a symbolic Tensor (cond_2/strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported</code></p>
  <p>I looked online and it suggests to downgrade the numpy version &lt; 1.20.0 (<a href="https://stackoverflow.com/questions/58479556/notimplementederror-cannot-convert-a-symbolic-tensor-2nd-target0-to-a-numpy">NotImplementedError: Cannot convert a symbolic Tensor (2nd_target:0) to a numpy array</a>). Note, the version numpy version must be &gt;= 1.19.2 for tensorflow 2.2.0.</p>
  <p>Scenario #2:</p>
  <ul>
  <li>tensorflow: 2.2.0</li>
  <li>numpy: 1.19.2-5</li>
  </ul>
  <p><code>ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject</code></p>
  <p>However, the online recommendation is to upgrade numpy to &gt;= 1.20.0. (<a href="https://stackoverflow.com/questions/66060487/valueerror-numpy-ndarray-size-changed-may-indicate-binary-incompatibility-exp">ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject</a>). There is a github issue related to this: <a href="https://github.com/tensorflow/models/issues/9749" rel="noreferrer">https://github.com/tensorflow/models/issues/9749</a>.</p>
  <p>I'm not sure what version to use to get the code running.</p>
  <p><strong>Appendix:
  Scenario #1 Error Code (numpy: 1.20.0-1)</strong></p>
  <pre><code>  File &quot;model_main_tf2.py&quot;, line 113, in &lt;module&gt;
      tf.compat.v1.app.run()
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\platform\app.py&quot;, line 40, in run
      _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\absl\app.py&quot;, line 303, in run
      _run_main(main, args)
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\absl\app.py&quot;, line 251, in _run_main
      sys.exit(main(argv))
    File &quot;model_main_tf2.py&quot;, line 104, in main
      model_lib_v2.train_loop(
    File &quot;C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\model_lib_v2.py&quot;, line 530, in train_loop
      train_input = strategy.experimental_distribute_datasets_from_function(
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\util\deprecation.py&quot;, line 340, in new_func
      return func(*args, **kwargs)
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\distribute\distribute_lib.py&quot;, line 1143, in experimental_distribute_datasets_from_function
      return self.distribute_datasets_from_function(dataset_fn, options)
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\distribute\distribute_lib.py&quot;, line 1134, in distribute_datasets_from_function
      return self._extended._distribute_datasets_from_function(  # pylint: disable=protected-access
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\distribute\mirrored_strategy.py&quot;, line 545, in _distribute_datasets_from_function
      return input_lib.get_distributed_datasets_from_function(
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\distribute\input_lib.py&quot;, line 161, in get_distributed_datasets_from_function
      return DistributedDatasetsFromFunction(dataset_fn, input_workers,
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\distribute\input_lib.py&quot;, line 1272, in __init__
      _create_datasets_from_function_with_input_context(
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\distribute\input_lib.py&quot;, line 1936, in _create_datasets_from_function_with_input_context
      dataset = dataset_fn(ctx)
    File &quot;C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\model_lib_v2.py&quot;, line 521, in train_dataset_fn
      train_input = inputs.train_input(
    File &quot;C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\inputs.py&quot;, line 893, in train_input
      dataset = INPUT_BUILDER_UTIL_MAP['dataset_build'](
    File &quot;C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\builders\dataset_builder.py&quot;, line 251, in build
      dataset = dataset_map_fn(dataset, decoder.decode, batch_size,
    File &quot;C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\builders\dataset_builder.py&quot;, line 236, in dataset_map_fn
      dataset = dataset.map_with_legacy_function(
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\util\deprecation.py&quot;, line 340, in new_func
      return func(*args, **kwargs)
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\data\ops\dataset_ops.py&quot;, line 2679, in map_with_legacy_function
      ParallelMapDataset(
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\data\ops\dataset_ops.py&quot;, line 4242, in __init__
      self._map_func = StructuredFunctionWrapper(
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\data\ops\dataset_ops.py&quot;, line 3493, in __init__
      self._function.add_to_graph(ops.get_default_graph())
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\framework\function.py&quot;, line 546, in add_to_graph
      self._create_definition_if_needed()
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\framework\function.py&quot;, line 378, in _create_definition_if_needed
      self._create_definition_if_needed_impl()
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\framework\function.py&quot;, line 400, in _create_definition_if_needed_impl
      temp_graph = func_graph_from_py_func(
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\framework\function.py&quot;, line 971, in func_graph_from_py_func
      outputs = func(*func_graph.inputs)
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\data\ops\dataset_ops.py&quot;, line 3485, in wrapper_fn
      ret = _wrapper_helper(*args)
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\data\ops\dataset_ops.py&quot;, line 3453, in _wrapper_helper
      ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
    File &quot;C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\autograph\impl\api.py&quot;, line 670, in wrapper
      raise e.ag_error_metadata.to_exception(e)
  NotImplementedError: in user code:
  
      C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\data_decoders\tf_example_decoder.py:524 default_groundtruth_weights  *
          [tf.shape(tensor_dict[fields.InputDataFields.groundtruth_boxes])[0]],
      C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\util\dispatch.py:201 wrapper  **
          return target(*args, **kwargs)
      C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\ops\array_ops.py:3120 ones
          output = _constant_if_small(one, shape, dtype, name)
      C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\ops\array_ops.py:2804 _constant_if_small
          if np.prod(shape) &lt; 1000:
      &lt;__array_function__ internals&gt;:5 prod
  
      C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\numpy\core\fromnumeric.py:3030 prod
          return _wrapreduction(a, np.multiply, 'prod', axis, dtype, out,
      C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\numpy\core\fromnumeric.py:87 _wrapreduction
          return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
      C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\framework\ops.py:852 __array__
          raise NotImplementedError(
  NotImplementedError: Cannot convert a symbolic Tensor (cond_2/strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported
  </code></pre>
  <p><strong>Scenario #2 Error Code (numpy: 1.19.2-5)</strong></p>
  <pre><code>2021-02-25 09:59:47.039105: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
  2021-02-25 09:59:47.044310: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
  Traceback (most recent call last):
    File &quot;model_main_tf2.py&quot;, line 32, in &lt;module&gt;
      from object_detection import model_lib_v2
    File &quot;C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\model_lib_v2.py&quot;, line 29, in &lt;module&gt;
      from object_detection import eval_util
    File &quot;C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\eval_util.py&quot;, line 35, in &lt;module&gt;
      from object_detection.metrics import coco_evaluation
    File &quot;C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\metrics\coco_evaluation.py&quot;, line 25, in &lt;module&gt;
      from object_detection.metrics import coco_tools
    File &quot;C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\metrics\coco_tools.py&quot;, line 51, in &lt;module&gt;
      from pycocotools import coco
    File &quot;C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\pycocotools\coco.py&quot;, line 56, in &lt;module&gt;
      from . import mask as maskUtils
    File &quot;C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\pycocotools\mask.py&quot;, line 3, in &lt;module&gt;
      import pycocotools._mask as _mask
    File &quot;pycocotools\_mask.pyx&quot;, line 1, in init pycocotools._mask
  ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject```
  </code></pre>