35 lines
761 B
Markdown
35 lines
761 B
Markdown
# Simple Transformer
|
|
|
|
A minimal Transformer implementation using NumPy.
|
|
|
|
## Files
|
|
|
|
- `transformer.py` - Core Transformer model implementation
|
|
- `example.py` - Usage example
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
python3 example.py
|
|
```
|
|
|
|
## Model Architecture
|
|
|
|
The implementation includes:
|
|
|
|
- **Multi-Head Attention**: Scaled dot-product attention with multiple heads
|
|
- **Feed-Forward Network**: Two-layer fully connected network with ReLU activation
|
|
- **Layer Normalization**: Applied after each sub-layer
|
|
- **Positional Encoding**: Sinusoidal position embeddings
|
|
|
|
## Model Parameters
|
|
|
|
Default configuration:
|
|
- Vocabulary size: 1000
|
|
- Model dimension: 512
|
|
- Number of heads: 8
|
|
- Number of layers: 6
|
|
- Feed-forward dimension: 2048
|
|
- Max sequence length: 100
|
|
|
|
Total parameters: ~19.4M |