Triangle104/Deepthink-Llama-3-8B-Preview-Q4_K_M-GGUF

This model was converted to GGUF format from prithivMLmods/Deepthink-Llama-3-8B-Preview using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

---

The Deepthink-Llama-3-8B-Preview is a fine-tuned version of the Llama-3.1-8B base model, further enhanced with the Rethinking R1 Dataset Logits for superior text generation. This model is designed for advanced reasoning, structured problem-solving, and contextually rich outputs, making it an excellent choice for applications in education, programming, research, and creative writing.

With its optimized architecture, Deepthink-Llama-3-8B-Preview excels at:

Logical reasoning and step-by-step problem solving
Mathematical and coding tasks, leveraging specialized expert models
Generating long-form content (up to 8K tokens) with improved coherence
Understanding structured data, including tables and JSON outputs
Instruction following and adapting to diverse system prompts, making it ideal for chatbots and AI assistants

    Key Features

Supports long-context processing of up to 128K tokens
Multilingual capabilities for 29+ languages, including English, Chinese, Spanish, French, German, Arabic, and more
Fine-tuned using Supervised Fine-Tuning (SFT) and Reinforcement Learning with Human Feedback (RLHF)

    Model Architecture

Deepthink-Llama-3-8B-Preview is built on the optimized transformer architecture of Llama-3.1-8B, integrating enhanced dataset logits from Rethinking R1 for better contextual understanding and output quality.

    Use with transformers

To run conversational inference using transformers >= 4.43.0, use the pipeline abstraction or leverage the generate() function with the Auto classes.

Ensure your environment is updated with:

pip install --upgrade transformers

    Example Usage

import torch from transformers import pipeline

model_id = "prithivMLmods/Deepthink-Llama-3-8B-Preview" pipe = pipeline( "text-generation", model=model_id, torch_dtype=torch.bfloat16, device_map="auto", )

messages = [ {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"}, {"role": "user", "content": "Who are you?"}, ]

outputs = pipe( messages, max_new_tokens=256, ) print(outputs[0]["generated_text"][-1])

    Intended Use

Deepthink-Llama-3-8B-Preview is designed for a wide range of applications requiring deep reasoning, structured outputs, and logical text generation. It is particularly suited for:

Education & Research: Generating detailed explanations, step-by-step solutions, and structured academic content.
Programming & Code Generation: Assisting in code writing, debugging, and algorithm explanations with improved logic structuring.
AI Chatbots & Assistants: Providing context-aware, instruction-following responses for conversational AI applications.
Creative Writing: Generating high-quality stories, articles, and structured narratives with coherence.
Data Analysis & Structured Output Generation: Interpreting and generating JSON, tables, and formatted outputs for structured data processing.

    Limitations

While Deepthink-Llama-3-8B-Preview is optimized for deep reasoning and structured outputs, it has some limitations:

Not a Real-time Knowledge Source

The model is trained on a fixed dataset and does not have real-time internet access. It may not provide up-to-date information on rapidly evolving topics.

Potential Biases

As with all AI models, responses may reflect biases present in the training data. Users should critically evaluate outputs, especially in sensitive domains.

Mathematical & Logical Reasoning Constraints

While strong in step-by-step reasoning, it may occasionally produce incorrect mathematical calculations or logical inconsistencies. External verification is recommended for critical applications.

Handling of Extremely Long Contexts

While it supports up to 128K tokens, efficiency and coherence may degrade when processing very long documents or conversations.

Limited Handling of Ambiguity

The model may struggle with highly ambiguous or context-dependent queries, sometimes generating plausible but incorrect responses.

Ethical & Compliance Considerations

Not intended for generating misinformation, automating legal or medical decisions, or other high-risk applications without human oversight.

---

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo Triangle104/Deepthink-Llama-3-8B-Preview-Q4_K_M-GGUF --hf-file deepthink-llama-3-8b-preview-q4_k_m.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo Triangle104/Deepthink-Llama-3-8B-Preview-Q4_K_M-GGUF --hf-file deepthink-llama-3-8b-preview-q4_k_m.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo Triangle104/Deepthink-Llama-3-8B-Preview-Q4_K_M-GGUF --hf-file deepthink-llama-3-8b-preview-q4_k_m.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo Triangle104/Deepthink-Llama-3-8B-Preview-Q4_K_M-GGUF --hf-file deepthink-llama-3-8b-preview-q4_k_m.gguf -c 2048