Training DistilGPT2, running into import error
Hello!I am trying to train distilgpt2 with my own data on colab. The problem is, even after installing the packages from requirements.txt, it is running into an import error. I am reproducing the entire error below.Traceback (most recent call last):File “train.py”, line 28, infrom distiller import DistillerFile “/content/transformers/examples/distillation/distiller.py”, line 31, infrom grouped_batch_sampler import GroupedBatchSampler, create_lengths_groupsFile “/content/transformers/examples/distillation/grouped_batch_sampler.py”, line 24, infrom utils import loggerFile “/content/transformers/examples/distillation/utils.py”, line 23, inimport gitFile “/usr/local/lib/python3.6/dist-packages/git/init.py”, line 38, infrom git.exc import *                       #@NoMove@IgnorePep8File “/usr/local/lib/python3.6/dist-packages/git/exc.py”, line 9, infrom git.compat import UnicodeMixin, safe_decode, string_typesFile “/usr/local/lib/python3.6/dist-packages/git/compat.py”, line 16, infrom gitdb.utils.compat import (ModuleNotFoundError: No module named ‘gitdb.utils.compat’Any help is appreciated. Is this because of an improper version of python?NB : I have restarted the runtime after installing packages from requirements.txt , but still the error persists.Thank you for your time!
2020-07-20T14:08:39Z
Are you sure you ranpip install -r requirements.txtin the distillation subfolder of the repo? It looks like you are missing some required packages.
2020-07-21T14:51:34Z
Yes. I have installed all packages using the given command and restarted the runtime to use them all.Is this a problem pertaining to google-colab?I am reproducing the output of the pip install requirements command.Requirement already satisfied: transformers in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 1)) (3.0.2)
Collecting gitpython==3.0.2
  Downloading https://files.pythonhosted.org/packages/8e/c7/70bd352e8a561a9b6d1cde9aa313b9d7c871b0c94c3821f44c01f3187e1d/GitPython-3.0.2-py3-none-any.whl (453kB)
     |████████████████████████████████| 460kB 6.3MB/s 
Requirement already satisfied: tensorboard>=1.14.0 in /usr/local/lib/python3.6/dist-packages (from -r requirements.txt (line 4)) (2.2.2)
Collecting tensorboardX==1.8
  Downloading https://files.pythonhosted.org/packages/c3/12/dcaf67e1312475b26db9e45e7bb6f32b540671a9ee120b3a72d9e09bc517/tensorboardX-1.8-py2.py3-none-any.whl (216kB)
     |████████████████████████████████| 225kB 16.4MB/s 
Collecting psutil==5.6.6
  Downloading https://files.pythonhosted.org/packages/5f/dc/edf6758183afc7591a16bd4b8a44d8eea80aca1327ea60161dd3bad9ad22/psutil-5.6.6.tar.gz (447kB)
     |████████████████████████████████| 450kB 10.7MB/s 
Collecting scipy==1.3.1
  Downloading https://files.pythonhosted.org/packages/29/50/a552a5aff252ae915f522e44642bb49a7b7b31677f9580cfd11bcc869976/scipy-1.3.1-cp36-cp36m-manylinux1_x86_64.whl (25.2MB)
     |████████████████████████████████| 25.2MB 129kB/s 
Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (20.4)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (1.18.5)
Requirement already satisfied: sacremoses in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (0.0.43)
Requirement already satisfied: tokenizers==0.8.1.rc1 in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (0.8.1rc1)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (2019.12.20)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (4.41.1)
Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (2.23.0)
Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (3.0.12)
Requirement already satisfied: sentencepiece!=0.1.92 in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (0.1.91)
Requirement already satisfied: dataclasses; python_version < "3.7" in /usr/local/lib/python3.6/dist-packages (from transformers->-r requirements.txt (line 1)) (0.7)
Collecting gitdb2>=2.0.0
  Downloading https://files.pythonhosted.org/packages/52/7e/59f96b47f671b3fe0aa0c1b609531a540434b719a10c417581e25b383909/gitdb2-4.0.2-py3-none-any.whl
Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (1.30.0)
Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (1.17.2)
Requirement already satisfied: wheel>=0.26; python_version >= "3" in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (0.34.2)
Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (3.12.2)
Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (1.12.0)
Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (0.9.0)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (1.7.0)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (0.4.1)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (1.0.1)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (3.2.2)
Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard>=1.14.0->-r requirements.txt (line 4)) (49.1.0)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers->-r requirements.txt (line 1)) (2.4.7)
Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers->-r requirements.txt (line 1)) (0.16.0)
Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers->-r requirements.txt (line 1)) (7.1.2)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers->-r requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers->-r requirements.txt (line 1)) (2.10)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers->-r requirements.txt (line 1)) (1.24.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers->-r requirements.txt (line 1)) (2020.6.20)
Collecting gitdb>=4.0.1
  Downloading https://files.pythonhosted.org/packages/48/11/d1800bca0a3bae820b84b7d813ad1eff15a48a64caea9c823fc8c1b119e8/gitdb-4.0.5-py3-none-any.whl (63kB)
     |████████████████████████████████| 71kB 9.5MB/s 
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=1.14.0->-r requirements.txt (line 4)) (0.2.8)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=1.14.0->-r requirements.txt (line 4)) (4.1.1)
Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3" in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=1.14.0->-r requirements.txt (line 4)) (4.6)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=1.14.0->-r requirements.txt (line 4)) (1.3.0)
Requirement already satisfied: importlib-metadata; python_version < "3.8" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard>=1.14.0->-r requirements.txt (line 4)) (1.7.0)
Collecting smmap<4,>=3.0.1
  Downloading https://files.pythonhosted.org/packages/b0/9a/4d409a6234eb940e6a78dfdfc66156e7522262f5f2fecca07dc55915952d/smmap-3.0.4-py2.py3-none-any.whl
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.6/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard>=1.14.0->-r requirements.txt (line 4)) (0.4.8)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=1.14.0->-r requirements.txt (line 4)) (3.1.0)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < "3.8"->markdown>=2.6.8->tensorboard>=1.14.0->-r requirements.txt (line 4)) (3.1.0)
Building wheels for collected packages: psutil
  Building wheel for psutil (setup.py) ... done
  Created wheel for psutil: filename=psutil-5.6.6-cp36-cp36m-linux_x86_64.whl size=274397 sha256=c857e5eafb2cdb184bdd46c7ec670c3baaa970e8ca8e2f619ede2e35657c66ed
  Stored in directory: /root/.cache/pip/wheels/d2/24/03/45099ba838d8bfe67d44d36581135a5b59b71c6c8620f5575a
Successfully built psutil
ERROR: tensorflow 2.2.0 has requirement scipy==1.4.1; python_version >= "3", but you'll have scipy 1.3.1 which is incompatible.
ERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.
Installing collected packages: smmap, gitdb, gitdb2, gitpython, tensorboardX, psutil, scipy
  Found existing installation: psutil 5.4.8
    Uninstalling psutil-5.4.8:
      Successfully uninstalled psutil-5.4.8
  Found existing installation: scipy 1.4.1
    Uninstalling scipy-1.4.1:
      Successfully uninstalled scipy-1.4.1
Successfully installed gitdb-4.0.5 gitdb2-4.0.2 gitpython-3.0.2 psutil-5.6.6 scipy-1.3.1 smmap-3.0.4 tensorboardX-1.8Thanks for your help!
2022-09-29T15:53:41Z
is there any command line interface available to do so.kindly pls suggest.and how to upload python files and folders… as in import feature.& even suggest how to write following command in requirements if cmd line interface is not availablegit clone -b opencv-2.4 --single-branch https://github.com/Itseez/opencv.git