Simontwice commited on
Commit
21a2f0a
·
verified ·
1 Parent(s): cd933a1

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +4 -2
README.md CHANGED
@@ -44,14 +44,16 @@ mistral-chat $HOME/mistral_models/mamba-codestral-7B-v0.1 --instruct --max_toke
44
  ```
45
 
46
  ## Evaluation
47
- We evaluate Codestral Mamba and open-weight models of the similar size on industry-standard benchmarks.
48
  | Benchmarks | HumanEval | MBPP | Spider | CruxE | HumanEval C++ | HumanEvalJava |HumanEvalJS |HumanEval Bash |
49
  | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
50
  | CodeGemma 1.1 7B | 61.0% | 67.7% | 46.3% | 50.4% | 49.1% | 41.8% | 52.2% | 9.4% |
51
  | CodeLlama 7B | 31.1% | 48.2% | 29.3% | 50.1% | 31.7% | 29.7% | 31.7% | 11.4% |
52
  | DeepSeek v1.5 7B | 65.9% | **70.8%** | **61.2%** | 55.5% | 59.0% | **62.7%** | 60.9% | **33.5%** |
53
  | **Codestral Mamba (7B)** | **75.0%** | 68.5% | 58.8% | **57.8%** | **59.8%** | 57.0% | **61.5%** | 31.1% |
54
-
 
 
55
 
56
  ## The Mistral AI Team
57
 
 
44
  ```
45
 
46
  ## Evaluation
47
+ We evaluate Codestral Mamba, Codestral and open-weight models of similar size on industry-standard benchmarks.
48
  | Benchmarks | HumanEval | MBPP | Spider | CruxE | HumanEval C++ | HumanEvalJava |HumanEvalJS |HumanEval Bash |
49
  | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
50
  | CodeGemma 1.1 7B | 61.0% | 67.7% | 46.3% | 50.4% | 49.1% | 41.8% | 52.2% | 9.4% |
51
  | CodeLlama 7B | 31.1% | 48.2% | 29.3% | 50.1% | 31.7% | 29.7% | 31.7% | 11.4% |
52
  | DeepSeek v1.5 7B | 65.9% | **70.8%** | **61.2%** | 55.5% | 59.0% | **62.7%** | 60.9% | **33.5%** |
53
  | **Codestral Mamba (7B)** | **75.0%** | 68.5% | 58.8% | **57.8%** | **59.8%** | 57.0% | **61.5%** | 31.1% |
54
+ |
55
+ | **Codestral (22B)** | **81.1%%** | **78.2%%** | **63.5%%** | 51.3% | **65.2%** | **63.3%** | - | **42.4%** |
56
+ | CodeLlama 34B | 43.3% | 75.1% | 50.8% | 55.2% | 51.6% | 57.0% | 59.0% | 29.7% |
57
 
58
  ## The Mistral AI Team
59