| language: | |
| - en | |
| license: other | |
| library_name: sglang | |
| pipeline_tag: text-generation | |
| tags: | |
| - grok-2 | |
| - xai | |
| - sglang | |
| - inference | |
| - triton | |
| base_model: xai-org/grok-2 | |
| model-index: | |
| - name: grok-2 | |
| results: [] | |
| # Grok 2 | |
| This repository contains the weights of Grok 2, a model trained and used at xAI in 2024. | |
| - License: Grok 2 Community License Agreement (./LICENSE) | |
| - Ownership: xAI (no changes to license or weights in this PR) | |
| ## Weights | |
| - Download from the Hub (≈500 GB total; 42 files): | |
| hf download xai-org/grok-2 --local-dir /local/grok-2 | |
| If you see transient errors, retry until it completes. | |
| ## Hardware and Parallelism | |
| - This checkpoint is configured for TP=8. | |
| - Recommended: 8× GPUs (each > 40 GB memory). | |
| ## Serving with SGLang (>= v0.5.1) | |
| Install SGLang from https://github.com/sgl-project/sglang/ | |
| Launch an inference server: | |
| python3 -m sglang.launch_server \ | |
| --model /local/grok-2 \ | |
| --tokenizer-path /local/grok-2/tokenizer.tok.json \ | |
| --tp 8 \ | |
| --quantization fp8 \ | |
| --attention-backend triton | |
| Send a test request (chat template aware): | |
| python3 -m sglang.test.send_one --prompt \ | |
| "Human: What is your name?<|separator|>\n\nAssistant:" | |
| You should see the model respond with its name: “Grok”. | |
| More ways to send requests: | |
| - https://docs.sglang.ai/basic_usage/send_request.html | |
| - Note: this is a post-trained model; use the correct chat template: | |
| https://github.com/sgl-project/sglang/blob/97a38.../tiktoken_tokenizer.py#L106 | |
| ## Community Usage (Examples) | |
| - Local-only serving behind VPN/Nginx allowlist | |
| - Log and audit inference (timestamps and SHA-256 manifests) | |
| - Optional cloud fallback to xAI’s API when local capacity is unavailable | |
| These are usage patterns only; they don’t alter license or weights. | |
| ## Limitations and Safety | |
| - Large memory footprint (multi-GPU recommended) | |
| - Follow the Grok 2 Community License | |
| - Redact any sensitive data before inference if routing via cloud services | |
| ## License | |
| Weights are licensed under the Grok 2 Community License Agreement (./LICENSE). | |
| خياراتك الآن | |