transformer_study/README.md

Simple Transformer

A minimal Transformer implementation using NumPy.

Files

• transformer.py - Core Transformer model implementation
• example.py - Usage example

Usage

python3 example.py

Model Architecture

The implementation includes:

• Multi-Head Attention: Scaled dot-product attention with multiple heads
• Feed-Forward Network: Two-layer fully connected network with ReLU activation
• Layer Normalization: Applied after each sub-layer
• Positional Encoding: Sinusoidal position embeddings

Model Parameters

Default configuration:

• Vocabulary size: 1000
• Model dimension: 512
• Number of heads: 8
• Number of layers: 6
• Feed-forward dimension: 2048
• Max sequence length: 100

Total parameters: ~19.4M