|
|
--- |
|
|
library_name: transformers |
|
|
license: cc-by-sa-4.0 |
|
|
language: |
|
|
- en |
|
|
- ja |
|
|
base_model: |
|
|
- EQUES/JPharmatron-7B-base |
|
|
tags: |
|
|
- pharmacy |
|
|
- biology |
|
|
- chemistry |
|
|
- medical |
|
|
--- |
|
|
|
|
|
# JPharmatron-7B |
|
|
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
|
|
JPharmatron-7B is a 7B large language model designed for pharmaceutical applications and researches. |
|
|
|
|
|
|
|
|
## Model Details |
|
|
|
|
|
### Model Description |
|
|
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
|
|
The JPharmatron-7B is continually pre-trained using 8.8B tokens from Japanese and English datasets, based on Qwen2.5-7B. Compared to the JPharmatron-7B-base model, JPharmatron-7B has enhanced chat capabilities, obtained from Qwen2.5-7B-Instruct's chat vector. |
|
|
|
|
|
- **Developed by:** EQUES Inc. |
|
|
- **Funded by [optional]:** [GENIAC Project](https://www.meti.go.jp/policy/mono_info_service/geniac/index.html) |
|
|
- **Model type:** Causal decoder-only |
|
|
- **Language(s) (NLP):** Japanese, English |
|
|
- **License:** CC-BY-SA-4.0 |
|
|
|
|
|
### Model Sources [optional] |
|
|
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
|
|
- **Repository:** https://github.com/EQUES-Inc/pharma-LLM-eval |
|
|
- **Paper [optional]:** [A Japanese Language Model and Three New Evaluation Benchmarks for Pharmaceutical NLP](https://arxiv.org/abs/2505.16661) |
|
|
|
|
|
## Uses |
|
|
|
|
|
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. --> |
|
|
|
|
|
This model is intended for applications in pharmaceutical paperwork and research. It is not validated for medical use or any other risk-sensitive use. |
|
|
|
|
|
|
|
|
## Evaluation |
|
|
|
|
|
<!-- This section describes the evaluation protocols and provides the results. --> |
|
|
|
|
|
We evaluated our model, JPharmatron-7B, with other general / domain-specific models of a similar size. |
|
|
|
|
|
### Testing Data |
|
|
|
|
|
<!-- This should link to a Dataset Card if possible. --> |
|
|
|
|
|
[JPharmaBench](https://huggingface.co/collections/EQUES/jpharmabench-680a34acfe96870e41d050d8) and two existing benchmarks (JMMLU (pharma) and IgakuQA) were used. |
|
|
|
|
|
### Results |
|
|
|
|
|
Compared to Meditron3-Qwen2.5-7B and Llama3.1-Swallow-8B-Instruct-v0.3, JPharmatron-7B achieved the highest score on all of the five benchmarks. |
|
|
|
|
|
 |
|
|
|
|
|
## Citation [optional] |
|
|
|
|
|
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. --> |
|
|
|
|
|
**BibTeX:** |
|
|
|
|
|
``` |
|
|
@misc{sukeda_japanese_2025, |
|
|
title = {A {Japanese} {Language} {Model} and {Three} {New} {Evaluation} {Benchmarks} for {Pharmaceutical} {NLP}}, |
|
|
url = {http://arxiv.org/abs/2505.16661}, |
|
|
doi = {10.48550/arXiv.2505.16661}, |
|
|
abstract = {We present a Japanese domain-specific language model for the pharmaceutical field, developed through continual pretraining on 2 billion Japanese pharmaceutical tokens and 8 billion English biomedical tokens. To enable rigorous evaluation, we introduce three new benchmarks: YakugakuQA, based on national pharmacist licensing exams; NayoseQA, which tests cross-lingual synonym and terminology normalization; and SogoCheck, a novel task designed to assess consistency reasoning between paired statements. We evaluate our model against both open-source medical LLMs and commercial models, including GPT-4o. Results show that our domain-specific model outperforms existing open models and achieves competitive performance with commercial ones, particularly on terminology-heavy and knowledge-based tasks. Interestingly, even GPT-4o performs poorly on SogoCheck, suggesting that cross-sentence consistency reasoning remains an open challenge. Our benchmark suite offers a broader diagnostic lens for pharmaceutical NLP, covering factual recall, lexical variation, and logical consistency. This work demonstrates the feasibility of building practical, secure, and cost-effective language models for Japanese domain-specific applications, and provides reusable evaluation resources for future research in pharmaceutical and healthcare NLP. Our model, codes, and datasets are released at https://github.com/EQUES-Inc/pharma-LLM-eval.}, |
|
|
urldate = {2025-05-30}, |
|
|
publisher = {arXiv}, |
|
|
author = {Sukeda, Issey and Fujii, Takuro and Buma, Kosei and Sasaki, Shunsuke and Ono, Shinnosuke}, |
|
|
month = may, |
|
|
year = {2025}, |
|
|
note = {arXiv:2505.16661 [cs]}, |
|
|
annote = {Comment: 15 pages, 9 tables, 5 figures} |
|
|
} |
|
|
|
|
|
``` |
|
|
|
|
|
## More Information [optional] |
|
|
|
|
|
See our preprint: [A Japanese Language Model and Three New Evaluation Benchmarks for Pharmaceutical NLP](https://arxiv.org/abs/2505.16661). |
|
|
|
|
|
## Model Card Authors [optional] |
|
|
|
|
|
[@shinnosukeono](https://shinnosukeono.github.io/) |
|
|
|
