CobrIX/CobrIX-1.0-Coder-Flash-33B-A13B

CobrIX-1.0-Coder-Flash-MoE

This is a Mixture-of-Experts (MoE) decoder built directly from the dense Qwen 3.5 base model (empero-ai/Qwythos-9B-v2) and 5 dense Qwen 3.5 fine-tuned experts, without relying on CobrIXKit.

The model features 33B total parameters, 13B active parameters (A13B) per token, and a native 1,048,576-token context window.

Architecture

• Model Class: Qwen35MoEForCausalLM (decoder-only), model_type=qwen35_moe.

• Context Window: Native support for up to 1,048,576 tokens (max_position_embeddings=1048576).

• Layer Design: Every transformer layer replaces the dense mlp with a sparse MoE block:

input_layernorm
    -> linear_attn
    -> post_attention_layernorm
    -> gate / experts[0..12] / shared_expert
    -> residual

• Routing: num_local_experts=5, num_experts_per_tok=2 (top-2 routing with softmax over the 5 experts).

• Shared Expert: An always-active shared expert based on the base MLP, gated by sigmoid(x @ w), with num_shared_experts=1.

• Precision: Router logits are computed in float32.

• Initialization: Router initialization is random. Shared-expert gates are initialized to zero.

Layer Types

layer_types = [
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention',
    'linear_attention', 'linear_attention', 'linear_attention', 'full_attention'
]

Weights

Base Model

The base model contributes:

• embed_tokens
• linear_attn.*
• Layer normalization parameters
• norm
• rotary_emb
• lm_head

Experts

The 5 expert models contribute only:

• gate_proj
• up_proven
• down_proj

for every transformer layer.

Integrity

No weight is modified, averaged, or interpolated during assembly.

The MoE model is constructed by preserving the original base-model components while routing the expert MLP projections through the sparse expert system.

Usage

The model directory contains self-contained:

• configuration_qwen35_moe.py
• modeling_qwen35_moe.py

These modules are loaded automatically through Hugging Face auto_map.

Therefore, no monkey patching is required.

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "CobrIX/CobrIX-1.0-Coder-Flash-33B-A13B"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    trust_remote_code=True
)

inputs = tokenizer(
    "Hello, how are you?",
    return_tensors="pt"
)

out = model.generate(
    **inputs,
    max_new_tokens=64
)

print(
    tokenizer.decode(
        out[0],
        skip_special_tokens=True
    )
)

Building & Verification

To replicate the build or verify the model structure, use the provided scripts.

Build the MoE Model

python convert.py \
    --base empero-ai/Qwythos-9B-v2 \
    --experts <e0> <e1> <e2> <e3> <e4> \
    --output Qwen35-CobrIX-MoE

Verify Architecture Integrity

python verify.py \
    --model Qwen35-CobrIX-MoE

Run Generation Tests

python test.py \
    --model Qwen35-CobrIX-MoE

Model Specifications

Specification	Value
Architecture	Qwen 3.5 MoE
Model Class	Qwen35MoEForCausalLM
Model Type	qwen35_moe
Total Parameters	~33B
Active Parameters	~13B (A13B)
Local Experts	5
Experts per Token	2
Shared Experts	1
Context Length	1,048,576 tokens
Attention Types	Linear + Full Attention
Router Precision	float32
Framework	PyTorch
Transformers	Hugging Face Transformers
License	MIT

Donations for Infrastructure

The development of custom AI architectures such as CobrIX-1.0-Coder-Flash-MoE requires ongoing computing resources, research, experimentation, and maintenance.

If this project brings value to your work or company, you can support the development and infrastructure of the CobrIX ecosystem through the wallets below.

All donations help fund:

• AI infrastructure
• GPU computing - Nvidia RTX PRO 6000 Blackwell 96 gb vram
• Open-source model development
• New model training
• Research and experimentation
• Community support

Bitcoin (BTC)

bc1q8mu8fjak4y84qj4dlk8pu4d3zhknm92zra4r4m

Ethereum (ETH / ERC-20)

0x8D9187dEa0a77390ef668361cd5b236DE54af2BB

Solana (SOL)

GQR2jZnWuWP1c3dbuz4mC7ZnyveacBKy63q8qf9nj8bp

Thank you to all developers, researchers, enthusiasts, and partners supporting the evolution of open and accessible artificial intelligence.

License

This project is released under the MIT License, subject to the licenses and terms of the underlying base model and expert models.

---

CobrIX — Open AI infrastructure and custom model research.