This repository contains a from-scratch implementation of a Transformer model in PyTorch for text classification, built without using any pre-defined Transformer models. The project follows the original Attention Is All You Need design principles and satisfies the assignment requirement of achieving >80% test accuracy.
- Task: Multi-class text classification
- Model: Encoder-only Transformer (self-attention + feed-forward blocks)
- Framework: PyTorch
- Dataset: AG News Topic Classification
- Best Test Accuracy: 91.25%
The goal of this project is to understand and implement every core component of a Transformer architecture, including embeddings, positional encoding, multi-head self-attention, residual connections, and layer normalization.
AG News Topic Classification Dataset
- Classes (4): World, Sports, Business, Sci/Tech
- Samples: 120,000 training + 7,600 test examples
- Type: News headlines and short descriptions
🔗 Dataset Links:
- Hugging Face: https://huggingface.co/datasets/ag_news
- Kaggle: https://www.kaggle.com/datasets/amananandrai/ag-news-classification-dataset
The following preprocessing steps were applied:
- Text cleaning (punctuation & special character removal)
- Lowercasing
- Tokenization
- Vocabulary construction
- Sequence padding/truncation (
max_len = 100) - Numerical encoding using word indices
- Pre-trained GloVe embeddings (50d) were used for initialization
- Source: https://nlp.stanford.edu/projects/glove/
The model is an encoder-only Transformer designed for classification tasks.
Input Text
↓
Token Embedding + Positional Encoding
↓
Transformer Encoder × N
↓
Global Average Pooling
↓
Fully Connected Layer
↓
Softmax (4 classes)
| Parameter | Value |
|---|---|
| Embedding Dim | 50 |
| Transformer Layers | 2 |
| Attention Heads | 2 |
| Feedforward Dim | 128 |
| Dropout | 0.1 |
| Max Sequence Len | 100 |
| Optimizer | Adam |
| Loss Function | Cross-Entropy |
- Train / Validation / Test Split: 80 / 10 / 10
- Batch Size: 32
- Epochs: up to 10 (Early Stopping applied)
- Regularization Techniques:
- Dropout
- Early stopping
- Learning rate tuning
| Metric | Value |
|---|---|
| Training Accuracy | 96.37% |
| Validation Accuracy | 91.96% |
| Test Accuracy | 91.25% |
| Test Loss | 0.2624 |
- Precision: 0.9135
- Recall: 0.9125
- F1-score: 0.9123
The model comfortably exceeds the assignment requirement of 80% test accuracy.
The notebook includes the following plots:
- Training vs Validation Accuracy
- Training vs Validation Loss
- Confusion Matrix
- ROC Curve (multi-class)
These plots help diagnose overfitting, convergence behavior, and class-wise performance.
- Best-performing model weights are saved and reloaded for final evaluation
- Ensures reproducibility and correct evaluation on the test set
# Clone the repository
git clone https://github.com/<your-username>/transformer-from-scratch-pytorch.git
cd transformer-from-scratch-pytorch
# Install dependencies
pip install -r requirements.txt
# Run the notebook
jupyter notebook Transformer_With_PyTorch.ipynb
⚠️ Note: GloVe embeddings (~800MB unzipped) are downloaded automatically inside the notebook.
.
├── Transformer_With_PyTorch.ipynb
├── README.md
├── assets/
│ └── transformer_architecture.png
├── saved_models/
│ └── best_transformer.pt
└── requirements.txt
- Vaswani et al., Attention Is All You Need, NeurIPS 2017
- PyTorch Transformer Documentation: https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoder.html
- Pennington et al., GloVe: Global Vectors for Word Representation
- AG News Dataset: https://huggingface.co/datasets/ag_news
This project was completed as part of CSE 676-B: Deep Learning at the University at Buffalo by Poojan Kaneriya (poojanra@buffalo.edu).
⭐ If you find this project helpful, feel free to star the repository!