Abstractive Summarization with BART (PyTorch + Hugging Face)

This repository contains an abstractive text summarization pipeline using a pre-trained encoder–decoder LLM (facebook/bart-base) fine-tuned on the BillSum dataset. The implementation follows the official Assignment 2 (Part IV) specification: fine-tune BART and evaluate using ROUGE, BLEU, and BERTScore.

---

📌 Project Highlights

• Task: Abstractive summarization of long-form legislative text
• Model: facebook/bart-base (Transformer encoder–decoder)
• Dataset: BillSum (US bills + human-written summaries)
• Framework: PyTorch + Hugging Face Transformers
• Metrics: ROUGE-1 / ROUGE-2 / ROUGE-L, BLEU, BERTScore

---

📊 Dataset

BillSum (Bills Summarization)

BillSum is a summarization dataset for US Congressional and California state bills, where the input is a legal document and the target is a human-written summary.

🔗 Dataset Link:

• Hugging Face: https://huggingface.co/datasets/billsum

---

🧹 Preprocessing

Text and summaries are tokenized using BartTokenizer:

• Max input length: 1024 tokens
• Max target length: 256 tokens
• Truncation enabled to fit within model limits

Tokenized datasets are stored locally using load_from_disk() to avoid repeated preprocessing.

---

🧠 Model

Base Model

• facebook/bart-base
• Encoder–decoder Transformer designed for sequence-to-sequence generation

🔗 Model card:

• https://huggingface.co/facebook/bart-base

---

🏋️ Fine-tuning Setup

The notebook fine-tunes BART using Hugging Face Trainer / Seq2SeqTrainer with generation enabled.

Training Configuration

Parameter	Value
Learning Rate	5e-5
Epochs	3
Train Batch Size	4
Eval Batch Size	4
Weight Decay	0.01
Evaluation Strategy	steps (eval_steps=100)
Generation Max Length	128
Checkpoint Saving	epoch (keep best 1)

Data Subset Used (for speed)

To make experimentation faster, the notebook trains and evaluates on a smaller subset:

• Train: 1000 samples
• Validation: 100 samples
• Test: 100 samples

---

📈 Results

The fine-tuned model achieves strong summarization performance on the test split.

ROUGE

Metric	Score
ROUGE-1	0.4778
ROUGE-2	0.2827
ROUGE-L	0.3507
ROUGE-Lsum	0.4064

BLEU

• BLEU: 19.87

BERTScore

• BERTScore (F1 avg): 0.8820

These results exceed the expected thresholds mentioned in the assignment spec for BillSum.

---

🔍 Evaluation

Evaluation is performed using the evaluate library:

• evaluate.load("rouge")
• evaluate.load("sacrebleu")
• evaluate.load("bertscore")

The notebook also prints example generated summaries vs reference summaries for qualitative inspection.

---

💾 Saving & Loading

The workflow supports saving the best checkpoint during training (save_strategy="epoch") and reloading it later for evaluation / inference.

---

▶️ How to Run

# Clone the repository
git clone https://github.com/<your-username>/bart-summarization-billsum.git
cd bart-summarization-billsum

# Install dependencies
pip install -r requirements.txt

# Run the notebook
jupyter notebook Summarization_using_LLMs.ipynb

---

🧪 Inference Example

Inside the notebook, you can generate a summary for any custom bill text:

1. Tokenize input text
2. Run model.generate(...)
3. Decode tokens using tokenizer.decode(...)

---

📁 Repository Structure

.
├── Summarization_using_LLMs.ipynb
├── README.md
├── requirements.txt

---

📚 References

1. Lewis et al., BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (ACL 2020)
2. Hugging Face BART model: https://huggingface.co/facebook/bart-base
3. BillSum dataset: https://huggingface.co/datasets/billsum
4. evaluate library: https://huggingface.co/docs/evaluate

---

🙌 Acknowledgements

This project was completed as part of CSE 676-B: Deep Learning at the University at Buffalo by Poojan Kaneriya (poojanra@buffalo.edu).

---

⭐ If this repo helped you, feel free to star it!