sahil_ramani / learn-gs
home writing projects notebooks about
12 notebooks
field_index // gaussian splatting

Gaussian Splatting from scratch

Twelve notebooks from project a point to a CUDA tile rasterizer with a training loop. Nothing is introduced until a problem in front of you forces it, so every notebook shows the break before the fix.

12 notebooks numpy › torch › cuda shipped executed github ↗

Every notebook here ships executed, so the figures and measured numbers are on the page without running anything. Colab opens a private copy on a Google VM that cannot affect this site or the repository; notebooks 10-12 ask it for a GPU and stop with instructions if they do not get one.

Phase A - CPU forward renderer (numpy)

notebook 01
points and why they fail

Pinhole camera, point cloud renderer. Breaks: points have no extent, surfaces turn to confetti.

notebook 02
the 2d gaussian primitive

A primitive with extent: the unnormalized 2D Gaussian. Covariance from R S, truncation, the 0.3 dilation and its energy bug.

notebook 03
3d gaussians and cameras

3D Gaussians (quaternion + scale). Monte Carlo shows perspective projection bends them: they stop being Gaussian.

notebook 04
projection ewa

The fix: linearize projection at the mean. Jacobian, Sigma2d = J W Sigma Wt Jt, where the approximation holds and where it visibly fails.

notebook 05
sort and composite

Many splats per pixel: the over operator, global depth sort, full renderer. Points vs splats side by side. Code promoted to src/gsplat_edu.

Phase B - real scenes

notebook 06
spherical harmonics

View-dependent color: flat RGB averages highlights away. SH basis degrees 0-3 with the ecosystem constants and signs, sheen demo, eval_sh promoted to the package.

notebook 07
render a real scene

A trained .ply is photographed reality in our exact representation. Layout pinned by a bit-exact round-trip, activations, batched conversions, auto-framed orbit on an honest CPU budget. Degrades to a synthetic stand-in when data/ is empty.

Phase C - training

notebook 08
fit an image 2d

Nothing chooses the Gaussians. Stripped to 2D: the over operator's backward recurrence derived by hand, finite-difference checked on every parameter, numpy Adam fits a procedural image. train/train_2d.py.

notebook 09
train 3d torch

The 3D chain, differentiable in PyTorch: autograd owns the gradients (gradcheck-verified), a random cloud becomes the sphere from 24 self-rendered views, densify and prune with every paper deviation named. The ms/iter it prints is phase D's opening argument. train/train_3d.py.

Phase D - CUDA

notebook 10
cuda naive

The measured CPU-vs-16ms gap. One thread per pixel, every thread reads every splat; parity with the numpy renderer, timing table, and the traffic arithmetic that convicts the design.

notebook 11
cuda tiled

Naive traffic scales as splats x pixels. 16x16 tiles, (tile << 32) | float_bits(depth) key sort, per-tile ranges, shared-memory batches, block-wide early exit. The actual 3DGS forward.

notebook 12
cuda backward

Fast kernels learn nothing. Forward stores final T + last contributor; backward replays notebook 08's recurrence per tile with atomics, wrapped in torch.autograd.Function; training at kernel speed. A minimal diff-gaussian-rasterization.