MikeRoz

/

Strawberrylemonade-L3-70B-v1.1-exl2

like 1

The quant is done and it's queued for upload, but there are a whole bunch of GLM-4.5 exl3 weights that are ahead of it.

Thanks Man.

Thanks for the quant just checked but i think it's broken. Then output is weird combination of words, other 3bpw 70b quants works fine. :\

Just checked and it works for me. Hashes on the weights match what's shown in HF too.

hmm.. it's giving geberish.

Works with tensor parallelism on or off, all the large file hashes match what I see on HuggingFace - not sure what else I can do.

(exl3venv) [REDACTED]@aventine:~/exllamav2/examples$ python chat.py -m [REDACTED]/sophosympatheia_Strawberrylemonade-L3-70B-v1.1-3.0bpw-h6-exl2 --mode llama3 -gs auto
 -- Model: [REDACTED]/sophosympatheia_Strawberrylemonade-L3-70B-v1.1-3.0bpw-h6-exl2
 -- Options: ['gpu_split: auto']
 -- Loading tokenizer...
 -- Loading model...
 -- Prompt format: llama3
 -- System prompt:

Assist users with tasks and answer questions to the best of your knowledge. Provide helpful and informative responses. Be conversational and engaging. If you are unsure or lack knowledge on a topic, admit it and try to find the answer or suggest where to find it. Keep responses concise and relevant. Follow ethical guidelines and promote a safe and respectful interaction.

User: What is the difference between a duck?

That's an interesting question! There are quite a few differences between ducks, actually. Here are some key distinctions:

1. Species: There are many different species of ducks, including Pekin, Mallard, Muscovy, Wood ducks, and others. Each has unique characteristics.

2. Size: Ducks vary greatly in size. For example, Muscovy ducks are relatively large (up to 3 feet long), while wood ducks are more compact (about 17 inches long).

3. Feathers: The color patterns, feather arrangements, and types of feathers differ between species. Some, like the Mallard, have vibrant green heads. Others, like the Wood duck, have striking multicolored patterns.

4. Beak shape: The shape and size of a duck's bill can vary. Some have stouter, broader bills for eating plants, while others have slimmer bills better suited for catching fish.

5. Habitat: While many ducks live in ponds and lakes, some prefer wetlands, rivers, or even saltwater environments. The Harlequin duck, for instance, spends much of its time in fast-moving streams.

6. Migration: Some duck species are migratory, while others remain in their year-round habitats. The Mallard tends to migrate, while the Muscovy duck generally does not.

7. Quacking: Believe it or not, ducks make different sounds! The Mallard has a distinctive "quack," while the Wood duck produces a whistling call.

8. Diet: While ducks generally eat plants, insects, and small animals, some species focus more on certain food sources. The Merganser duck, for example, specializes in catching fish.

These are just a few of the many differences between ducks. They're fascinating creatures with a lot of variety in their family!

User: ^CTraceback (most recent call last):
  File "/home/[REDACTED]/exllamav2/examples/chat.py", line 311, in <module>
    content = input().strip()
              ^^^^^^^
KeyboardInterrupt

(exl3venv) [REDACTED]@aventine:~/exllamav2/examples$ python chat.py -m [REDACTED]/sophosympatheia_Strawberrylemonade-L3-70B-v1.1-3.0bpw-h6-exl2 --mode llama3 -tp
 -- Model: [REDACTED]/sophosympatheia_Strawberrylemonade-L3-70B-v1.1-3.0bpw-h6-exl2
 -- Options: ['tensor_parallel']
 -- Loading tokenizer...
 -- Loading model...
 -- Prompt format: llama3
 -- System prompt:

Assist users with tasks and answer questions to the best of your knowledge. Provide helpful and informative responses. Be conversational and engaging. If you are unsure or lack knowledge on a topic, admit it and try to find the answer or suggest where to find it. Keep responses concise and relevant. Follow ethical guidelines and promote a safe and respectful interaction.

User: What is the difference between a duck?

You didn't finish your question about ducks. What specific difference are you wondering about? Ducks differ from other birds in several ways:

1. Plumage: Ducks have special feathers that trap air next to their skin, keeping them warm in cold water. Their feathers are also coated with oils that make them very water-repellent.

2. Legs and feet: Ducks have webbed feet, which help them swim and dive. Their legs are usually short and strong.

3. Beak shape: Duck beaks are typically broad and flat, which helps them forage for plants and small animals in the water.

4. Habitat: Most ducks live in or near water, such as ponds, lakes, rivers, or wetlands.

5. Diet: While different duck species eat different things, many ducks primarily eat plants, insects, and small crustaceans.

If you had a specific comparison in mind (like ducks vs. geese vs. swans), please let me know and I can provide more details!

User: ^CTraceback (most recent call last):
  File "/home/[REDACTED]/exllamav2/examples/chat.py", line 311, in <module>
    content = input().strip()
              ^^^^^^^
KeyboardInterrupt

(exl3venv) [REDACTED]@aventine:~/exllamav2/examples$ sha256sum [REDACTED]/sophosympatheia_Strawberrylemonade-L3-70B-v1.1-3.0bpw-h6-exl2/*.safetensors
c04287e470cf88aa2e57c8a4297b3232c9bf9b599172ccac67e2e21020baa47d  [REDACTED]/sophosympatheia_Strawberrylemonade-L3-70B-v1.1-3.0bpw-h6-exl2/output-00001-of-00004.safetensors
a0cebe1c70cd5455b7d370906e0b63dd3efb4acd26ea5eed97df1fa4bab79c47  [REDACTED]/sophosympatheia_Strawberrylemonade-L3-70B-v1.1-3.0bpw-h6-exl2/output-00002-of-00004.safetensors
56a6c40f213e98c84432d3ae3533d192f9ce7bdfa64f6cfe42e5030d918c11fc  [REDACTED]/sophosympatheia_Strawberrylemonade-L3-70B-v1.1-3.0bpw-h6-exl2/output-00003-of-00004.safetensors
cebdb11a2b05ce348efda8a77a3ccd1ad38deebc03b94fbe64924db4453b9628  [REDACTED]/sophosympatheia_Strawberrylemonade-L3-70B-v1.1-3.0bpw-h6-exl2/output-00004-of-00004.safetensors
(exl3venv) [REDACTED]@aventine:~/exllamav2/examples$ sha256sum [REDACTED]/sophosympatheia_Strawberrylemonade-L3-70B-v1.1-3.0bpw-h6-exl2/tokenizer.json
6b9e4e7fb171f92fd137b777cc2714bf87d11576700a1dcd7a399e7bbe39537b  [REDACTED]/sophosympatheia_Strawberrylemonade-L3-70B-v1.1-3.0bpw-h6-exl2/tokenizer.json
(exl3venv) [REDACTED]@aventine:~/exllamav2/examples$

bruh. i tried other models 3bpw they're working fine. :\ can you send me your tabbyapi config?

In the examples above I didn't use TabbyAPI, I used examples/chat.py from the exllamav2 repo, direct link here. Sometimes integration with backends can be the issue, so when troubleshooting I always fall back to chat.py as a minimal example.

kk