|
--- |
|
datasets: |
|
- bigcode/the-stack-v2 |
|
base_model: |
|
- microsoft/codebert-base |
|
--- |
|
# grammarBERT |
|
|
|
`grammarBERT` is a specialized fine-tuning of `codeBERT`, using a Masked Language Modeling (MLM) task focused on derivation sequences specific to Python 3.8. By fine-tuning on Python’s Abstract Syntax Tree (AST) structures, `grammarBERT` combines `codeBERT`’s capabilities in natural language and code token handling with a unique focus on derivation sequences, enhancing performance for grammar-based programming tasks. This is particularly useful for applications requiring syntactic understanding, improved parsing accuracy, and context-aware code generation or transformation. |
|
|
|
## Model Overview |
|
|
|
- **Base Model**: `codeBERT` |
|
- **Task**: Masked Language Modeling on derivation sequences |
|
- **Supported Language**: Python 3.8 |
|
- **Applications**: Parsing, code transformation, syntactic analysis, grammar-based programming |
|
|
|
## Model Usage |
|
|
|
To use the `grammarBERT` model with Python 3.8-specific derivation sequences, load the model and tokenizer as shown below: |
|
|
|
```python |
|
from transformers import RobertaForMaskedLM, RobertaTokenizer |
|
|
|
# Load the pre-trained grammarBERT model and tokenizer |
|
model = RobertaForMaskedLM.from_pretrained("Nbeau/grammarBERT") |
|
tokenizer = RobertaTokenizer.from_pretrained("Nbeau/grammarBERT") |
|
|
|
# Tokenize and prepare a code snippet |
|
code_snippet = "def enumerate_items(items):" |
|
# Convert code to a derivation sequence (requires `ast2seq` function) |
|
derivation_sequence = ast2seq(code_snippet) # `ast2seq` available at https://github.com/NathanaelBeau/grammarBERT/asdl/ |
|
input_ids = tokenizer.encode(derivation_sequence, return_tensors='pt') |
|
|
|
# Use the model for masked token prediction or further fine-tuning |
|
outputs = model(input_ids) |
|
``` |
|
|
|
### Training and Fine-Tuning |
|
|
|
To train your own `grammarBERT` on a custom dataset or adapt it for different Python versions, follow the setup instructions in the [grammarBERT GitHub repository](https://github.com/NathanaelBeau/grammarBERT). The repository provides detailed guidance for: |
|
|
|
- Preparing Python Abstract Syntax Tree (AST) sequences. |
|
- Configuring tokenization for derivation sequences. |
|
- Running training scripts for Masked Language Modeling (MLM) fine-tuning. |
|
|
|
This setup allows for targeted fine-tuning on derivation sequences tailored to your specific grammar requirements. |
|
|
|
|
|
|
