Self-Supervised Learning · Model Compression

Distill DINOv2
into 2.8M
parameters

Train strong visual representations on a single CPU in 30 minutes. No GPU. No labels. No infrastructure.

View on GitHub → Read Paper →
2.8M
97 %+
Accuracy Retained
7×
Fewer Parameters
$1.50
Training Cost

Built independently
by a single researcher.

Emran Abdu
Independent Researcher
github.com/Emran-goat
@Emran_py
LinkedIn
jakeniel98@gmail.com
01 Results

Downstream task performance

Flowers102
96.3%
98.5% of DINOv2-S/14 accuracy
Oxford Pets
92.1%
97.4% of DINOv2-S/14 accuracy
EuroSAT
97.6%
99.5% of DINOv2-S/14 accuracy
BreastMNIST
79.8%
96.8% of DINOv2-S/14 accuracy
02 Method

Three-objective distillation

01

Masked Image Modeling

Mask 75% of patches. The student predicts teacher features at masked positions through a JEPA objective — no pixel reconstruction, no negative pairs.

02

Feature Alignment

Cosine similarity between student and teacher CLS tokens. Keeps the global representation consistent while the student learns its own tokenizer.

03

KoLeo Regularization

Uniformity constraint on the feature space. Prevents collapse and encourages diverse, well-distributed representations without momentum encoders.

— Architecture

How the distillation pipeline flows

Input
Images
224×224 patches
→
Teacher (frozen)
DINOv2-S/14
22M params
→
Student (trained)
TinySSL
2.8M params
→
Output
Features
256-dim vectors
03 Get Started

Three lines to inference

# Install
pip install tinyssl

# Load a pre-trained model
import torch
from tinyssl.models.students import TinySSLBase

model = TinySSLBase.from_pretrained(
    "tinyssl-base-flowers102"
)
model.eval()

# Extract features
features = model(images) # → (B, 256)

Install. Load. Infer.

TinySSL ships with pre-trained weights for Flowers102, Oxford Pets, EuroSAT, and BreastMNIST.

  • ONNX export for production deployment
  • CoreML conversion for Apple Silicon
  • TorchScript for mobile inference
  • 12MB model size — fits anywhere
04 Cost

Training cost comparison

DINOv2-S/14
$1,000,000+
22M params
TinySSL-Base
$1.50
2.8M params
MAE ViT-B
$30,000+
86M params
SimCLR
$15,000+
28M params

Start training
on your laptop.

No GPU required. No labels needed. Just your CPU and a coffee.

View on GitHub → Read the Paper → Hugging Face →