Initial upload of Gemma3 SigLIP vision encoder (code + weights)

.gitattributes

@@ -0,0 +1,2 @@
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text

NOTICE

@@ -0,0 +1,2 @@
+Gemma is provided under and subject to the Gemma Terms of Use
+found at https://ai.google.dev/gemma/terms

README.md

@@ -0,0 +1,88 @@
+---
+license: gemma
+tags:
+- image-feature-extraction
+- siglip
+base_model: google/gemma-3-4b-pt
+library_name: transformers
+---
+
+# Gemma 3 Vision Encoder (extracted)
+
+This repository contains the SigLIP-family vision encoder extracted from **google/gemma-3-4b-pt**.
+It also includes the Gemma multimodal projector weights (state_dict) and a small metadata file.
+
+## Contents
+
+- `config.json`, `model.safetensors`: the SigLIP vision encoder
+- `preprocessor_config.json`: the image processor settings used by Gemma 3
+- `projector_state_dict.pt`: PyTorch state dict for the Gemma projector
+- `projector_config.json`: metadata (class, dims, token count if detected)
+- `NOTICE`: Gemma Terms pointer
+
+## Basic usage (encoder as feature extractor)
+
+```python
+from transformers import SiglipVisionModel, AutoImageProcessor
+from PIL import Image
+import torch
+
+repo_id = "<your-username>/<your-repo>"
+encoder = SiglipVisionModel.from_pretrained(repo_id).eval()
+processor = AutoImageProcessor.from_pretrained(repo_id)
+
+img = Image.open("test.jpg").convert("RGB")
+inputs = processor(images=img, return_tensors='pt')
+with torch.no_grad():
+    feats = encoder(**inputs).last_hidden_state  # (B, Tv, Dv)
+print(feats.shape)
+```
+
+## Using the projector (Gemma-style multimodal path)
+
+The projector here is provided as a **state dict** plus metadata. It is intended for users
+who are wiring a Gemma-style VLM, where the projector maps the vision sequence to a fixed number
+of image tokens at the LLM hidden size.
+
+Two common paths:
+
+1) **Use with Transformers' Gemma 3 model**: load the full VLM, then load this projector's state_dict
+   into the model's `multi_modal_projector` module.
+
+```python
+import torch
+from transformers import Gemma3ForConditionalGeneration
+
+repo_id = "<your-username>/<your-repo>"
+vlm = Gemma3ForConditionalGeneration.from_pretrained('google/gemma-3-4b-pt', device_map='cpu')
+sd = torch.load('projector_state_dict.pt', map_location='cpu')  # or from the repo checkout
+vlm.multi_modal_projector.load_state_dict(sd, strict=False)
+vlm.eval()
+```
+
+2) **Recreate the projector module from the class name**, instantiate it, and load the state dict.
+   The metadata file records the fully qualified class name (FQN).
+
+```python
+import importlib, json, torch
+
+with open("projector_config.json", "r") as f:
+    meta = json.load(f)
+fqn = meta.get('projector_fqn')  # e.g., 'transformers.models.gemma3.modeling_gemma3.Gemma3VisionProjector'
+mod_name, cls_name = fqn.rsplit('.', 1)
+cls = getattr(importlib.import_module(mod_name), cls_name)
+projector = cls(**{k: v for k, v in meta.items() if k.endswith('_dim') or k.endswith('_tokens')})
+sd = torch.load('projector_state_dict.pt', map_location='cpu')
+projector.load_state_dict(sd, strict=False)
+projector.eval()
+```
+
+## Shapes (for reference)
+- Vision hidden size Dv: 1152
+- Projector output tokens Ti: 256
+- Projector class: `transformers.models.gemma3.modeling_gemma3.Gemma3MultiModalProjector`
+
+## License / Terms
+
+See the `NOTICE` file. Gemma is provided under and subject to the Gemma Terms of Use:
+https://ai.google.dev/gemma/terms

config.json

@@ -0,0 +1,19 @@
+{
+  "architectures": [
+    "SiglipVisionModel"
+  ],
+  "attention_dropout": 0.0,
+  "hidden_act": "gelu_pytorch_tanh",
+  "hidden_size": 1152,
+  "image_size": 896,
+  "intermediate_size": 4304,
+  "layer_norm_eps": 1e-06,
+  "model_type": "siglip_vision_model",
+  "num_attention_heads": 16,
+  "num_channels": 3,
+  "num_hidden_layers": 27,
+  "patch_size": 14,
+  "torch_dtype": "float16",
+  "transformers_version": "4.54.1",
+  "vision_use_head": false
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ee46a32cef11c6e19af11edd1b75fa153113c066a893ff3797f9356c628dfc0b
+size 833785376

preprocessor_config.json

@@ -0,0 +1,29 @@
+{
+  "do_convert_rgb": null,
+  "do_normalize": true,
+  "do_pan_and_scan": null,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "Gemma3ImageProcessor",
+  "image_seq_length": 256,
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "pan_and_scan_max_num_crops": null,
+  "pan_and_scan_min_crop_size": null,
+  "pan_and_scan_min_ratio_to_activate": null,
+  "processor_class": "Gemma3Processor",
+  "resample": 2,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 896,
+    "width": 896
+  }
+}

projector_config.json

@@ -0,0 +1,6 @@
+{
+  "projector_fqn": "transformers.models.gemma3.modeling_gemma3.Gemma3MultiModalProjector",
+  "vision_hidden_dim_Dv": 1152,
+  "llm_hidden_dim_H": null,
+  "output_tokens_Ti": 256
+}

projector_state_dict.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3791a79c06b0e1438e3bc9998c6ac10d37a9bbdb63e273ae2c0a682ff233e5ba
+size 5902246