Update README.md

README.md

@@ -109,6 +109,112 @@ The fine-tuning dataset was compiled from the following sources:
     *   `save_steps`: 1000
 *   **Optimization Kernel:** Liger kernel enabled (`use_liger=True`) for increased throughput and reduced memory usage via optimized Triton kernels for common LLM operations.
 
+## Inference - vLLM
+
+### Installation
+
+```bash
+!pip -q install vllm
+!pip -q install bitsandbytes>=0.45.3
+```
+
+### vLLM Inference
+
+```python
+import os
+import torch
+from vllm import LLM
+from vllm import SamplingParams
+
+os.environ['VLLM_USE_V1'] = '0'
+
+model_id = "aiplanet/LuxLlama"
+llm = LLM(model=model_id, dtype=torch.bfloat16, trust_remote_code=True, max_model_len=8192)
+
+prompts = [
+"""
+<system>
+You are a highly capable assistant trained to handle a variety of tasks. Below is an instruction that describes a task, paired with an input that provides further context. Your goal is to provide an accurate, clear, and contextually appropriate response that fulfills the user's request.
+</system>
+<user>
+### Instruction:
+Berechent d'Resultat vun dëser Rechnung.
+
+### Input:
+Wéi vill ass (12 + 8) × 2 - 10?
+</user>
+
+<assistant>
+""",
+"""
+<system>
+You are a highly capable assistant trained to handle a variety of tasks. Below is an instruction that describes a task, paired with an input that provides further context. Your goal is to provide an accurate, clear, and contextually appropriate response that fulfills the user's request.
+</system>
+<user>
+### Instruction:
+Iwwersetz dësen Saz vum Engleschen an d’Lëtzebuergesch.
+
+### Input:
+'We are going to the market tomorrow morning.'
+</user>
+
+<assistant>
+""",
+"""
+<system>
+You are a highly capable assistant trained to handle a variety of tasks. Below is an instruction that describes a task, paired with an input that provides further context. Your goal is to provide an accurate, clear, and contextually appropriate response that fulfills the user's request.
+</system>
+<user>
+### Instruction:
+Beäntwert d’Fro mat gesonde Mënscheverstand.
+
+### Input:
+Kann een engem Bësch eng Email schécken? Firwat oder firwat net?
+</user>
+
+<assistant>
+""",
+]
+# Create a sampling params object.
+sampling_params = SamplingParams(temperature=0.1, top_p=0.1, top_k=1, max_tokens=1024)
+
+def main(llm):
+    # Create an LLM.
+    llm = llm
+    # Generate texts from the prompts.
+    # The output is a list of RequestOutput objects
+    # that contain the prompt, generated text, and other information.
+    outputs = llm.generate(prompts, sampling_params)
+    # Print the outputs.
+    print("\nGenerated Outputs:\n" + "-" * 60)
+    for output in outputs:
+        prompt = output.prompt
+        generated_text = output.outputs[0].text
+        print(f"Prompt:    {prompt!r}")
+        print(f"Output:    {generated_text!r}")
+        print("-" * 60)
+
+
+if __name__ == "__main__":
+    main(llm)
+```
+
+#### Output
+
+```bash
+Generated Outputs:
+------------------------------------------------------------
+Prompt:    "\n<system>  \nYou are a highly capable assistant trained to handle a variety of tasks. Below is an instruction that describes a task, paired with an input that provides further context. Your goal is to provide an accurate, clear, and contextually appropriate response that fulfills the user's request.  \n</system>  \n<user>  \n### Instruction:  \nBerechent d'Resultat vun dëser Rechnung.  \n\n### Input:  \nWéi vill ass (12 + 8) × 2 - 10?\n</user>  \n\n<assistant>\n"
+Output:    "### Response:\n D'Resultat vun der Rechnung ass 30. Hei ass d'Schrëtt-fir-Schrëtt Berechnung: 1. Füügt 12 an 8: 12 + 8 = 20 2. Multiplizéiert d'Resultat mat 2: 20 × 2 = 40 3. Subtrahéiert 10 vum Resultat: 40 - 10 = 30\n</assistant>  \n"
+------------------------------------------------------------
+Prompt:    "\n<system>  \nYou are a highly capable assistant trained to handle a variety of tasks. Below is an instruction that describes a task, paired with an input that provides further context. Your goal is to provide an accurate, clear, and contextually appropriate response that fulfills the user's request.  \n</system>  \n<user>  \n### Instruction:  \nIwwersetz dësen Saz vum Engleschen an d’Lëtzebuergesch.  \n\n### Input:  \n'We are going to the market tomorrow morning.'\n</user>  \n\n<assistant>\n"
+Output:    "### Response:\n 'Mir ginn muer fréi op de Maart.'\n</assistant>  \n"
+------------------------------------------------------------
+Prompt:    "\n<system>  \nYou are a highly capable assistant trained to handle a variety of tasks. Below is an instruction that describes a task, paired with an input that provides further context. Your goal is to provide an accurate, clear, and contextually appropriate response that fulfills the user's request.  \n</system>  \n<user>  \n### Instruction:  \nBeäntwert d’Fro mat gesonde Mënscheverstand. \n\n### Input:  \nKann een engem Bësch eng Email schécken? Firwat oder firwat net?\n</user>  \n\n<assistant>\n"
+Output:    "### Response:\n Et ass net méiglech eng E-Mail un e Bësch ze schécken. E-Mail ass eng Form vu Kommunikatioun déi fir d'Mënschen entwéckelt gouf, an et erfuerdert eng digital Infrastruktur fir ze funktionéieren, wéi Internet a Computeren. Bëscher, op der anerer Säit, sinn natierlech Liewensraim, déi aus Planzen, Déieren an aner Elementer besteet. Si hunn keng Fäegkeet fir digital Kommunikatioun oder Informatioun ze verstoen, also kann keng E-Mail un e Bësch geschéckt ginn.\n</assistant>  \n"
+------------------------------------------------------------
+```
+
 ## Evaluation
 
 ### Benchmarking Dataset - LUXELLA
@@ -166,6 +272,7 @@ The fine-tuning dataset was compiled from the following sources:
 
 **Summary:** LuxLlama demonstrates strong performance on the LUXELLA benchmark, outperforming other tested models significantly. It excels in translation, comprehension, and verb conjugation. Areas like vocabulary, spelling, and idioms show relatively lower scores, indicating room for improvement in capturing finer linguistic nuances. The model handles beginner-level tasks very well, with a gradual decrease in performance as difficulty increases, validating the benchmark's sensitivity. Sample high-performing questions show correct handling of cultural knowledge, spelling, and advanced verb conjugation, while low-performing samples highlight challenges with specific grammar rules (Konjunktiv II usage), subtle distinctions in vocabulary (Niess vs Kusinn), and standard word order conventions.
 
+
 ## Learnings and Observations
 
 *(This section should be updated after further analysis and usage.)*