Projects

Three papers replicated tensor by tensor on Bengali data — no nn.Transformer, no nn.MultiheadAttention, no pre-built HuggingFace models. Every one trained end to end from random weights on a single Apple M1 with 16 GB of RAM, and measured against published baselines on identical splits.

1. BERT

   papers-from-scratch

   BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding·Devlin et al., 2019

   A 7.5M-parameter Bengali BERT, pre-trained from nothing on a laptop, that scores above published mBERT and IndicBERT on Bengali news classification.

   Params

   ~7.5M

   Corpus

   ~114MB

   Pre-trained

   ~28h

   Accuracy

   86.5%

   • Pre-trained a 7.5M-parameter BERT from scratch on a 114 MB Bengali Wikipedia corpus — MLM + NSP objectives, ~28 hours on laptop MPS.
   • Hand-wrote the three-way input embeddings, GELU feed-forward, MLM and NSP heads, dynamic 80/10/10 masking and the joint loss.
   • 86.5% test accuracy on 6-class Bengali news topics — above published mBERT (80.2, at 110M params) and IndicBERT (78.5), ~1 point under XLM-R at 17× fewer parameters.
   • PyTorch
   • Python
   BlogSource

2. Transformer

   papers-from-scratch

   Attention Is All You Need·Vaswani et al., 2017

   The original transformer rebuilt component by component, then trained to translate English into Bengali on one laptop.

   Params

   ~11M

   Paper

   ~65M

   Hardware

   M1, 16GB

   Data

   Samanantar

   • Built every component by hand — multi-head attention, sinusoidal positional encoding, label-smoothed loss, Noam schedule, weight tying and pooled beam search.
   • Scaled the paper's 65M-parameter base to 11M to fit 16 GB while keeping every method exact, and reported the resulting BLEU openly against the ~6.6× training-token gap rather than hiding it.
   • PyTorch
   • Python
   BlogSource

3. Vision Transformer

   papers-from-scratch

   An Image Is Worth 16x16 Words·Dosovitskiy et al., 2021

   A ViT trained on 1,935 photographs of Bengali terracotta temples — a dataset I had to build first — to test the paper's claim about pre-training. It failed, exactly as predicted.

   Images

   1,935

   Classes

   10

   From scratch

   14.9%

   Pre-trained ViT-B/16

   88.9%

   • Assembled a 1,935-image, 10-class dataset of Bengali terracotta temples and used it to test the paper's central claim.
   • Trained from scratch on those 1,935 images the ViT reached 14.9% — barely above the 10% chance line. Swapping in torchvision's pre-trained ViT-B/16 as a frozen feature extractor, same data, reached 88.9%. The 88.9% belongs to torchvision's weights, not to the from-scratch model; the gap between the two numbers is the paper's central claim, reproduced.
   • PyTorch
   • Python
   BlogSource