Initial commit: Simple Transformer implementation

2026-07-16 14:29:12 +08:00
commit 836060ecdf
4 changed files with 293 additions and 0 deletions
@@ -0,0 +1,35 @@
+# Simple Transformer
+
+A minimal Transformer implementation using NumPy.
+
+## Files
+
+- `transformer.py` - Core Transformer model implementation
+- `example.py` - Usage example
+
+## Usage
+
+```bash
+python3 example.py
+```
+
+## Model Architecture
+
+The implementation includes:
+
+- **Multi-Head Attention**: Scaled dot-product attention with multiple heads
+- **Feed-Forward Network**: Two-layer fully connected network with ReLU activation
+- **Layer Normalization**: Applied after each sub-layer
+- **Positional Encoding**: Sinusoidal position embeddings
+
+## Model Parameters
+
+Default configuration:
+- Vocabulary size: 1000
+- Model dimension: 512
+- Number of heads: 8
+- Number of layers: 6
+- Feed-forward dimension: 2048
+- Max sequence length: 100
+
+Total parameters: ~19.4M