How to set model_path to another directory?
Hello! I am trying to follow these instructions to have both GPU and CPU offloadinghuggingface.coQuantize 🤗 Transformers modelsWe’re on a journey to advance and democratize artificial intelligence through open source and open science.Below is the code I am usingSet the quantization config with llm_int8_enable_fp32_cpu_offload set to Truequantization_config = BitsAndBytesConfig(llm_int8_enable_fp32_cpu_offload=True)device_map = {“transformer.word_embeddings”: 0,“transformer.word_embeddings_layernorm”: 0,“lm_head”: “cpu”,“transformer.h”: 0,“transformer.ln_f”: 0,}model_path = “decapoda-research/llama-7b-hf”model_8bit = AutoModelForCausalLM.from_pretrained(model_path,device_map=device_map,quantization_config=quantization_config,)my problem is that it asks for the model_path to link to hugging face and a specific model. I also have these models downloaded but in another path:C:\Windows\System32\text-generation-webui\modelsWhen I run the code above it downloads the model again. How do I get the model_path to look at my already downloaded models?Even when I let the download run, I get this errorTraceback (most recent call last):File “C:\Windows\System32\text-generation-webui\server7b.py”, line 33, inmodel_8bit = AutoModelForCausalLM.from_pretrained(File “C:\Users\justi\miniconda3\envs\textgen\lib\site-packages\transformers\models\auto\auto_factory.py”, line 471, in from_pretrainedreturn model_class.from_pretrained(File “C:\Users\justi\miniconda3\envs\textgen\lib\site-packages\transformers\modeling_utils.py”, line 2643, in from_pretrained) = cls._load_pretrained_model(File “C:\Users\justi\miniconda3\envs\textgen\lib\site-packages\transformers\modeling_utils.py”, line 2966, in _load_pretrained_modelnew_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(File “C:\Users\justi\miniconda3\envs\textgen\lib\site-packages\transformers\modeling_utils.py”, line 662, in _load_state_dict_into_meta_modelraise ValueError(f"{param_name} doesn’t have any device set.")ValueError: model.layers.0.self_attn.q_proj.weight doesn’t have any device set.
2023-06-12T17:49:16Z
TheAIGuyz:File “C:\Users\justi\miniconda3\envs\textgen\lib\site-packages\transformers\modeling_utils.py”, line 662, in _load_state_dict_into_meta_modelraise ValueError(f"{param_name} doesn’t have any device set.")ValueError: model.layers.0.self_attn.q_proj.weight doesn’t have any device set.looks like your devicemap isn’t covering something called model.layers.0.self_attn.q_proj.weightLets make a net maybeCreate the device map dictionary with default device assignment to catch anything not specifieddevice_map = {param.name: default_device for param in model.parameters()}Assign specific devices to parametersdevice_map.update({“transformer.word_embeddings”: torch.device(“cuda:0”),“transformer.word_embeddings_layernorm”: torch.device(“cuda:0”),“lm_head”: torch.device(“cpu”),“transformer.h”: torch.device(“cuda:0”),“transformer.ln_f”: torch.device(“cuda:0”)})if you want to get specific,maybe something like this::from transformers import AutoModelForCausalLMmodel_path = “decapoda-research/llama-7b-hf”model = AutoModelForCausalLM.from_pretrained(model_path, local_files_only=True)Get the names of the model parametersparam_names = [param.name for param in model.parameters()]Print the parameter namesfor name in param_names:print(name)iterating overmodel.parameters()to find each parameter and retrieve itsnameattribute. This will provide you with the names of the model’s parameters for device assignmentTheAIGuyz:When I run the code above it downloads the model again. How do I get the model_path to look at my already downloaded models?Maybe have it check for the file first:import osfrom transformers import AutoModelForCausalLMmodel_name = “llama-7b-hf”model_directory = “models”Check if the model file exists in the specified directorymodel_file = os.path.join(model_directory, model_name)if os.path.exists(model_file):model_path = model_fileprint(“Model file found in the directory. Using the local model file.”)else:model_path = model_nameprint(“Model file not found in the directory. Downloading the model from the repository.”)Load the modelmodel = AutoModelForCausalLM.from_pretrained(model_path)