# Simple Transformer

A minimal Transformer implementation using NumPy.

## Files

- `transformer.py` - Core Transformer model implementation
- `example.py` - Usage example

## Usage

```bash
python3 example.py
```

## Model Architecture

The implementation includes:

- **Multi-Head Attention**: Scaled dot-product attention with multiple heads
- **Feed-Forward Network**: Two-layer fully connected network with ReLU activation
- **Layer Normalization**: Applied after each sub-layer
- **Positional Encoding**: Sinusoidal position embeddings

## Model Parameters

Default configuration:
- Vocabulary size: 1000
- Model dimension: 512
- Number of heads: 8
- Number of layers: 6
- Feed-forward dimension: 2048
- Max sequence length: 100

Total parameters: ~19.4M