Upload folder using huggingface_hub
Browse files
README.md
CHANGED
|
@@ -37,4 +37,42 @@ for token, id in zip(tokens, token_ids):
|
|
| 37 |
print(f"Token: {token:20} ID: {id}")
|
| 38 |
|
| 39 |
print(f"\nGenerated text:\n{generated_text}")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
```
|
|
|
|
| 37 |
print(f"Token: {token:20} ID: {id}")
|
| 38 |
|
| 39 |
print(f"\nGenerated text:\n{generated_text}")
|
| 40 |
+
```
|
| 41 |
+
|
| 42 |
+
Output:
|
| 43 |
+
```text
|
| 44 |
+
Token: Ċ ID: 198
|
| 45 |
+
Token: <tool_call> ID: 128013
|
| 46 |
+
Token: Ċ ID: 198
|
| 47 |
+
Token: {' ID: 13922
|
| 48 |
+
Token: arguments ID: 16774
|
| 49 |
+
Token: ': ID: 1232
|
| 50 |
+
Token: Ġ{' ID: 5473
|
| 51 |
+
Token: location ID: 2588
|
| 52 |
+
Token: ': ID: 1232
|
| 53 |
+
Token: Ġ' ID: 364
|
| 54 |
+
Token: Boston ID: 65432
|
| 55 |
+
Token: ', ID: 518
|
| 56 |
+
Token: Ġ' ID: 364
|
| 57 |
+
Token: unit ID: 3928
|
| 58 |
+
Token: ': ID: 1232
|
| 59 |
+
Token: Ġ' ID: 364
|
| 60 |
+
Token: f ID: 69
|
| 61 |
+
Token: ahrenheit ID: 49010
|
| 62 |
+
Token: '}, ID: 25762
|
| 63 |
+
Token: Ġ' ID: 364
|
| 64 |
+
Token: name ID: 609
|
| 65 |
+
Token: ': ID: 1232
|
| 66 |
+
Token: Ġ' ID: 364
|
| 67 |
+
Token: get ID: 456
|
| 68 |
+
Token: _current ID: 11327
|
| 69 |
+
Token: _weather ID: 70464
|
| 70 |
+
Token: 'Ċ ID: 1270
|
| 71 |
+
Token: </tool_call> ID: 128014
|
| 72 |
+
|
| 73 |
+
Generated text:
|
| 74 |
+
|
| 75 |
+
<tool_call>
|
| 76 |
+
{'arguments': {'location': 'Boston', 'unit': 'fahrenheit'}, 'name': 'get_current_weather'
|
| 77 |
+
</tool_call>
|
| 78 |
```
|