Create README.md (#1)

- Create README.md (693de0bcd85881d6fcb50df209f2fde9894c08eb)

README.md

@@ -0,0 +1,148 @@
+---
+license: apache-2.0
+base_model:
+- mistralai/Mistral-7B-Instruct-v0.3
+pipeline_tag: text2text-generation
+---
+
+# Elastic models
+
+Elastic models are the models produced by TheStage AI ANNA: Automated Neural Networks Accelerator.
+ANNA allows you to control model size, latency and quality with a simple slider movement. For each model, ANNA produces a series of optimized models:
+
+* __XL__: Mathematically equivalent neural network, optimized with our DNN compiler. 
+
+* __L__: Near lossless model, with less than 1% degradation obtained on corresponding benchmarks.
+
+* __M__: Faster model, with accuracy degradation less than 1.5%.
+
+* __S__: The fastest model, with accuracy degradation less than 2%.
+
+
+__Goals of elastic models:__
+
+* Provide flexibility in cost vs quality selection for inference
+* Provide clear quality and latency benchmarks
+* Provide interface of HF libraries: transformers and diffusers with a single line of code
+* Provide models supported on a wide range of hardware, which are pre-compiled and require no JIT.
+
+> It's important to note that specific quality degradation can vary from model to model. For instance, with an S model, you can have 0.5% degradation as well.
+
+
+## Inference
+
+To infer our models, you just need to replace `transformers` import with `elastic_models.transformers`:
+
+```python
+import torch
+from transformers import AutoTokenizer
+from elastic_models.transformers import AutoModelForCausalLM
+
+model_name = "mistralai/Mistral-7B-Instruct-v0.3"
+token = ''
+
+device = torch.device("cuda")
+tokenizer = AutoTokenizer.from_pretrained(model_name, token=token)
+
+model = AutoModelForCausalLM.from_pretrained(
+    model_name, 
+    token=token,
+    cache_dir="/mnt/rnd/huggingface_cache",
+    torch_dtype=torch.bfloat16,
+    attn_implementation="sdpa"
+).to(device)
+model.generation_config.pad_token_id = tokenizer.eos_token_id
+
+prompt = "Describe basics of DNNs quantization."
+inputs = tokenizer(prompt, return_tensors="pt")
+inputs.to(device)
+
+generate_ids = model.generate(**inputs, max_length=500)
+input_len = inputs['input_ids'].shape[1]
+generate_ids = generate_ids[:, input_len:]
+output = tokenizer.batch_decode(
+    generate_ids,
+    skip_special_tokens=True, 
+    clean_up_tokenization_spaces=False
+)[0]
+print(f"# Q:\n{prompt}\n")
+print(f"# A:\n{output}\n")
+```
+
+---
+### System requirements
+
+__GPUs__: H100, L40s
+
+__OS__: Linux #TODO
+
+__Python__: 3.10-3.12
+
+
+---
+### Installation
+
+```shell
+pip install thestage
+pip install elastic_models
+```
+
+Then go to app.thestage.ai, login and generate API token from your profile page.
+Set up API token as follows:
+
+```shell
+thestage config set --api-token <YOUR_API_TOKEN>
+```
+
+Congrats, now you can use accelerated models!
+
+
+## Benchmarks
+
+Benchmarking is one of the most important procedures during model acceleration. We aim to provide clear performance metrics for models using our algorithms. The `W8A8, int8 column` indicates that we applied W8A8 quantization with int8 data type to all linear layers and used the same calibration data as for ANNA. The S model achieves practically identical speed but much higher quality, as ANNA knows how to improve quantization quality on sensitive layers!
+
+### Quality benchmarks
+
+For quality evaluation we have used: #TODO link to github
+
+| Metric/Model  | S | M | L | XL | Original | W8A8, int8 |
+|---------------|---|---|---|----|----------|------------|
+| MMLU          | 0 | 0 | 0 | 0  | 0        | 0          |
+| PIQA          | 0 | 0 | 0 | 0  | 0        | 0          |
+| Arc Challenge | 0 | 0 | 0 | 0  | 0        | 0          |
+| Winogrande    | 0 | 0 | 0 | 0  | 0        | 0          |
+
+
+> __MMLU__: Evaluates/shows ... 
+
+> __MMLU__: Evaluates/shows ... 
+
+> __Arc Challenge__: Evaluates/shows ...
+
+> __PIQA__: Evaluates/shows ... 
+
+
+### Latency benchmarks
+
+We have profiled models in different scenarios:
+
+__100 input/300 output tok/s__:
+| GPU/Model | S   | M | L | XL | Original | W8A8, int8 |
+|-----------|-----|---|---|----|----------|------------|
+| H100      | 189 | 0 | 0 | 0  | 48       | 0          |
+| L40s      | 79  | 0 | 0 | 0  | 42       | 0          |
+
+
+__1000 input/1000 output tok/s__:
+| GPU/Model | S   | M | L | XL | Original | W8A8, int8 |
+|-----------|-----|---|---|----|----------|------------|
+| H100      | 189 | 0 | 0 | 0  | 48       | 0          |
+| L40s      | 79  | 0 | 0 | 0  | 42       | 0          |
+
+
+## Links
+
+* __Platform__: [app.thestage.ai](app.thestage.ai)
+* __Elastic models Github__: [app.thestage.ai](app.thestage.ai)
+* __Subscribe for updates__: [TheStageAI X](https://x.com/TheStageAI)
+* __Contact email__: [email protected]