Initial commit

.gitattributes

@@ -0,0 +1,38 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.json filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.txt filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,98 @@
+---
+license: mit
+---
+
+
+# FaceLLM-38B
+
+Multimodal large language models (MLLMs) have shown remarkable performance in vision-language tasks. However, existing MLLMs are primarily trained on generic datasets, limiting their ability to reason on domain-specific visual cues such as those in facial images. In particular, tasks that require detailed understanding of facial structure, expression, emotion, and demographic features remain underexplored by MLLMs due to the lack of large-scale annotated face image-text datasets. In this work, we introduce FaceLLM, a multimodal large language model trained specifically for facial image understanding. Our experiments demonstrate that FaceLLM achieves the state-of-the-art performance of MLLMs on various face-centric tasks.
+Project page: https://www.idiap.ch/paper/facellm
+
+![](imgs/overall_acc.png)
+
+## Overview
+
+* **Training**: FaceLLM, a Multimodal Large Language Model for Face Understanding
+* **Backbone**: InternVL3-1B
+* **Parameters**: 0.9B
+* **Task**: Face Understanding tasks, including bias and fairness (age estimation, gender prediction, race estimation), face recognition (high-resolution face recognition, low-resolution face recognition, celebrity identification), face authentication (face anti-spoofing, deepfake detection), face analysis (attributes prediction, facial expression
+recognition, headpose estimation), face localization (crowd counting, face parsing), etc.
+* **Framework**: Pytorch/Huggingface
+
+## Evaluation of Models
+*Comparison with MLLMs on the FaceXBench benchmark. The best performing model in each category is emboldened and the best model amongst all MLLMs is in purple.*
+
+![](imgs/benchmark.png)
+
+*Performance of FaceLLM models (FaceLLM-1B, FaceLLM-8B, and FaceLLM-38B) on different sub-tasks, including age estimation, gender prediction, race estimation, high-resolution face recognition, low-resolution face recognition, celebrity identification, face anti-spoofing, deepfake detection, attributes prediction, facial expression recognition, headpose estimation, face localization crowd counting, face parsing, and face tools retrieval.*
+
+![](imgs/FaceLLM.png)
+
+## Running Code
+
+Minimal code to instantiate the model and perform inference:
+
+``` python
+from transformers import AutoProcessor, AutoModelForImageTextToText
+import torch
+import os
+
+# Path to FaceLLM model folder
+model_path = "./FaceLLM-38B/model"  
+
+# Load processor and model
+processor = AutoProcessor.from_pretrained(model_path)
+model = AutoModelForImageTextToText.from_pretrained(model_path,
+                torch_dtype=torch.bfloat16,
+                low_cpu_mem_usage=True,
+                trust_remote_code=True)
+model.eval() # Set model to evaluation mode
+model.to("cuda") # Move model to GPU if available
+
+# Example for image and text input
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "image": "image.jpg"},
+            {"type": "text", "text": "Describe this image."},
+        ],
+    },
+]
+
+# Apply the chat template and tokenize
+inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device, dtype=torch.bfloat16)
+
+# Generate output
+generate_ids = model.generate(**inputs, max_new_tokens=500)
+
+# Decode the generated text
+decoded_output = processor.decode(generate_ids[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
+print(decoded_output)
+```
+
+## License
+
+This project is released under the MIT License. This project uses the pre-trained Qwen2.5 as a component, which is licensed under the Qwen License.
+
+
+## Copyright 
+
+Copyright (c) 2025, Hatef Otroshi Shahreza, Sébastien Marcel, Idiap Research Institute, Martigny 1920, Switzerland.
+
+https://www.idiap.ch/paper/facellm/
+
+Please refer to the link for information about the License & Copyright terms and conditions.
+
+## Citation
+
+If you find our work useful, please cite the following publication:
+
+```bibtex
+@article{facellm2025,
+  author    = {Hatef Otroshi Shahreza and S{\'e}bastien Marcel},
+  title     = {FaceLLM: A Multimodal Large Language Model for Face Understanding},
+  journal   = {arXiv preprint arXiv:2507.10300},
+  year      = {2025}
+}
+```

model/Modelfile

@@ -0,0 +1,16 @@
+# ollama modelfile auto-generated by llamafactory
+
+FROM .
+
+TEMPLATE """{{ if .System }}<|im_start|>system
+{{ .System }}<|im_end|>
+{{ end }}{{ range .Messages }}{{ if eq .Role "user" }}<|im_start|>user
+{{ .Content }}<|im_end|>
+<|im_start|>assistant
+{{ else if eq .Role "assistant" }}{{ .Content }}<|im_end|>
+{{ end }}{{ end }}"""
+
+SYSTEM """你是书生·万象，英文名是InternVL，是由上海人工智能实验室、清华大学及多家合作单位联合开发的多模态大语言模型。"""
+
+PARAMETER stop "<|im_end|>"
+PARAMETER num_ctx 4096

model/added_tokens.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:222d504fd2fb42e08a43b7ef6383a130483d34233f506b6ccc799655de48dff8
+size 811

model/chat_template.jinja

@@ -0,0 +1,6 @@
+{% for message in messages %}{{'<|im_start|>' + message['role'] + '
+'}}{% if message['content'] is string %}{{ message['content'] }}{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' %}{{ '<IMG_CONTEXT>
+' }}{% elif content['type'] == 'video' %}{{ '<video>
+' }}{% elif content['type'] == 'text' %}{{ content['text'] }}{% endif %}{% endfor %}{% endif %}{{'<|im_end|>
+'}}{% endfor %}{% if add_generation_prompt %}{{'<|im_start|>assistant
+' }}{% endif %}

model/config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f9c73fbb9d788b6b5b4fba2956c4a5506e9b51f6312038aa867d35567c2e4833
+size 2029

model/generation_config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d3c1b12ce6fece499a9fbadf05cd4adf133c616cba0fefd86cb6f1d5fc74dee
+size 121

model/merges.txt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8831e4f1a044471340f7c0a83d7bd71306a5b867e95fd870f74d0c5308a904d5
+size 1671853

model/model-00001-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c49905b7c0ed1d390c8c6a215cf6c84f698478c890a6fa7302213a3f74c6e0c
+size 993704064

model/model-00002-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:876a6972bb5fcaeafc73c94517b36fdf515174a35d15ab4e1a3c4bdd8613a05a
+size 983354568

model/model-00003-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:10d0c163d86b7af6313d5bf907bb9e3f9b0ff469585cbe0b479551117b1f827c
+size 983354592

model/model-00004-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b1c77d7080eca761af3af02e9dbfaf1d906d066c5894a1936da5590a03b8c08d
+size 983354624

model/model-00005-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e533170bf4128093e53d234ea8fa66342cb9da02fd772deba125ebb864c30d2a
+size 983354624

model/model-00006-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f7bf287c40b0dfbf8220ab78dafc6e433cb444b7e9bcff17c2ffe2446bdd9aa1
+size 983354624

model/model-00007-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c871b6dfc2553855118b7099c37c5debe95b3b5bec395affc5e8d04555c8debf
+size 983354624

model/model-00008-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9676975fdeb0a1c289b705aa9b9d698d20a5407a2036d26c83802294d14fa679
+size 983354624

model/model-00009-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:88cbb6eb0bd2652efa6988c2fe17986b9e3d2c638e4fa678765460c715379e3d
+size 983354624

model/model-00010-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f31d3e5dc76e623e9ce4df75f32cd3ae11c50ca47b2dd5382f97b2b1df16b6c
+size 983354624

model/model-00011-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:232c039bfda2073cd62e774dc68db78278566db8383161bdc8e9be116cbf2be4
+size 983354624

model/model-00012-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f8ef8f3d0f16e2111fa2d9df92af372716695cba6245882bd03fcd672cf10a2c
+size 1553141912

model/model-00013-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba7c41da341c6f49d525d89c86254b18c51f920aa8b630b9fcef2862d8fd7bb1
+size 838358808

model/model-00014-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67a38d5ad54c085bce0cb0ed57b24cc18beb1b55c398c85af6566534a7c648f8
+size 975212048

model/model-00015-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8ae8bd6d863ff7f9b70ded7b8e7758771b77765818f20ae8bb0877a67911a055
+size 975212048

model/model-00016-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:35c0158c54b5ba4ce4cf94abe21baef10b56c3ef510079071f0df8e8fd243b08
+size 975212048

model/model-00017-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:15c58819207566e31fdedfb5debd76cd4cf65c71b38932f3f4c738a92fb19389
+size 975212048

model/model-00018-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b34e3d4e7b4b8e105450a13fa7140f903f639fc8bde0814b67ed6f87721c04d
+size 975212048

model/model-00019-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff070d360e2ac818c3a0ed3cc65acf6e8629f71db955d0877b28585b6a72e39b
+size 975212048

model/model-00020-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c6687e84013d6c1aee18de94994969aaff04f25bc0ef1e6d223f02520c2564bd
+size 975212048

model/model-00021-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:88e35269ed254bac532c31e94ede4c9c924146b60276d03de621a212d7659d3f
+size 975212048

model/model-00022-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f8e5717d5e4a442a2c711615bbca8630ff4c5a96c3709f30bcb0cdd6dcad3a43
+size 975212048

model/model-00023-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5032dff99b399c78a045d2f8cc0279eca576551daf7d3f279187b734aab6316a
+size 975212064

model/model-00024-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0596ae7170263d0ced77e3d3a741fc009dcf2b800ed87aaa3eeccdb45591c27b
+size 975212064

model/model-00025-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c0294ab30a9454119285e3bb36f8f175126160abb973f6aea00b19191ed045dd
+size 975212064

model/model-00026-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d374f7e3ee2613610923b40c300c3e2471858e521d638b67a2608bd3d9f3b814
+size 975212064

model/model-00027-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b82c2c2c4f0cf177e1a0be8a8db9852929e9e52dfa9d1a88d6244f870fb33bf6
+size 975212064

model/model-00028-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c3fb48c82b11826d9a5ec8e0b8c23172c73e35ee6f47b61709d1e725f3da8a50
+size 975212064

model/model-00029-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3abee36bd5cd33dfb43f4b766948f7a6495f1e28aa8c7a6d696498ff0343dca5
+size 975212064

model/model-00030-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:16adcd71c6516607b2c218041ba52da14c3f390ce7790984451f7f6443f4494b
+size 975212064

model/model-00031-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0dfa9899741ead746b89aae6aa4c6a524215255eb234393a30e14930cdaf581b
+size 975212064

model/model-00032-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:514eedc4361a96080664a34a9ed0d396a73400092d931153770dfc85c4473699
+size 975212064

model/model-00033-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8279ae9ad744a7046cc0984fe0d24ed151af160d65a21e114056d5006a069261
+size 975212064

model/model-00034-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ffbcafd2ec6b306de04ebc8a2dcca72282849bd74214964b3efb025263c33f3
+size 975212064

model/model-00035-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a8e9598bf5957de794397c3ac0aea467fae599673e513271222f55421965f0f0
+size 975212064

model/model-00036-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:03287291771cef8ceb616116663f6881700d303d64379abb2db83768010f3876
+size 975212064

model/model-00037-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6fc9466d7afe5e974995a3674aca8853cbaa92fca4cb8d3aadbae9fe54f42df2
+size 975212064

model/model-00038-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6682bae8ee68b82847f67c9ec67e4e08fe32c32b46b99a82bebe53321daafca2
+size 975212064

model/model-00039-of-00078.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f21cbb4f620e352bd3b7a2b3837041ddcd3f027b48ca3cd4ab03e68ddeca6cf
+size 975212064