Kenton Kwok

Salsa and Bachata scene in Cambridge

2026-01-29T12:00:00+00:00

Surprise! After a few months of searching, I’ve discovered the existence of a Salsa/ Bachata scene in Cambridge (UK)! It’s mostly Cuban salsa and Bachata, but I’ve managed to find a nice bunch dancing crossbody.

Just wanted to share this online in case anyone else finds it useful.

Salsa

Monday is Cambridge University Cuban Salsa Society at Revs with Randy Despaigne and other teachers. There are usually 3 levels at the class 20:00-21:00 and the social lasts until 22:00.
Tuesday is Darwin Cuban Salsa at Downing Place starting at 20:30 with Randy Despaigne, Verónica del Tono and other teachers. There are 4 levels followed by a social. Tickets are released Sundays at 1900. Members of Darwin College Cambridge get a discount.
Wednesday is Cuban Salsa and Bachata classes followed by a social, at Downing Place, with Leandro Charanga starting at 1930
Thursday is Cambridge Cuban Salsa with Sacha from 1930 at the Meadows Community Centre, 299 Arbury Rd, Cambridge CB4 2JL.
Friday & Mondays are salsa classes with Club Salsa (LA on 1 style, not Cuban) at St Andrews Baptist Church at 2000 then a social until midnight. Salsa and Bachata are played.
Sunday is Cambridge Cuban Salsa with Sacha from 1930 at at Adélie Studios, 121 Chesterton Road, CB4 3AT.

Bachata

Mondays at Cambourne Church Center and Tuesdays in Hidden Rooms with Veronica and team at Bachata Cambridge. Bachata sensual is the main style.

Other dance

For University students, there unfortunately isn’t a Salsa/ Bachata team, but the Cambridge University Dancesports society does offer classes and does organise competitions! https://www.cambridgedancers.org/teams/cudt/

Technical Note: Beer Lambert Law for Oxygenation Extraction

2026-01-07T12:00:00+00:00

The most common approach in the literature for extracting oxygenation from multispectral or hyperspectral images, based on diffuse reflectance spectroscopy, is to use the Beer Lambert law.

Although the model is technically incorrect, it can give results that make sense.

Forward model– the Beer Lambert Law

This is a physics-based law (also used in atmospheric physics, metrology), that comes from summing together infinitesimal absorptions over distance. It describes the resultant intensity of light $I_T$ after passing through a medium (in transmission), basically saying that a unit increase in absorbance will be amplified exponentially as it continuously interacts with the medium over a distance.

This makes sense, as if there is a constant interaction probability per distance, this will be amplified with a thicker medium.

\[I_T(z) = I_0 \exp\bigg(-\int_0^z\mu_a(z') dz'\bigg)\]

To make it useful for us in diffuse reflectance, we assume that

light is attenuated by tissue according to the Beer lambert law, over an interaction distance $l$
all non-attenuated light is therefore backscattered and forms part of the diffuse reflectance
This is equivalent to saying, incorrectly, that so we can directly use Beer Lambert by assuming the transmitted light is the reflected light $I_T = I_R$
it is super incorrect but it is useful

We can make the assumptions, with non-interacting chromophores, that that the absorption coefficient is a linear combination of chromophores

The integral inside the bracket is the absorbance $a$

\[a(\lambda) = -\ln[I_{R}(\lambda)/I_{0}(\lambda)]=\sum_{i}\varepsilon_i (\lambda) c_{i}l+U\]

where

$a$ is the absorbance (dimensionless)
absorbers indexed by $i$
$\mu_a$ is the absorbance [cm-1]
(mean) path length $l$ [cm],
- $\varepsilon$ is the molar absorption coefficient [L mol-1 cm-1]
- $c$ is the molar concentration [mol L-1]
$U$ is a wavelength-independent ‘fudge factor’ that captures other losses

Assuming there are two dominant chromophores, oxy and deoxyhaemoglobin, we can proceed with our analysis. This is the molar extinction coefficient of oxy and deoxyhaemoglobin in water, compiled by Scott Prahl (Oregon Institute of Technology). This can be found here.

Graphs showing the molar extinction coefficients on a linear and logarithmic scale

Notes

A high extinction coefficient means that more light is absorbed in that wavelength
Deoxyhaemoglobin absorbs more in the reds by a (few) orders of magnitude, whilst oxyhaemoglobin absorbs slightly more in the greens

Simulation

We can put this all in a simulation to observe how changing the various parameters changes the resultant reflectance spectrum. We define the oxygenation as the fraction of oxygenated to deoxyhaemoglobin in

\[S_{O_2} = \frac{c_{HbO_2}}{c_{HbO_2 }+c_{HbO }}\]

Interactive plot showing the effects of varying different model parameters

Inverse: retrieval

By doing model-based retrieval, we mean to solve for the mixing coefficients (or the concentrations). By using a least squares solver, we obtain a point estimates of the quantities $(c_{HbO_2 }, c_{HbO }, U)$. Unless $l$ is fixed, it is absorbed into the concentrations.

We can observe several potential failure modes of the retrieval algorithm by playing around with the simulation.

When the path length is high, most of the useful signal is in the wavelengths above 600 nm. Increasing the scattering term has the same effect on the reflectance as reducing the oxygenation saturation.
Increasing the path length for a high oxygenation sample has the same effects as a reduction in the oxygenation saturation.

References

[1] N. T. Clancy, G. Jones, L. Maier-Hein, D. S. Elson, and D. Stoyanov, ‘Surgical spectral imaging’, Medical Image Analysis, vol. 63, p. 101699, Jul. 2020, doi: 10.1016/j.media.2020.101699.

Travelling abroad for work!

2025-08-30T12:00:00+00:00

Travel is eye opening and you can learn an awful deal just by getting out of a day-to-day routine.

I used to think that scientists/ engineers are only good in the lab or working behind a desk. This is far from the case! A lot of scientists do field work/ experimental collaborations abroad and attend meetings internationally. For me, I moved around different cities for jobs and in my previous role in industry I attended 2 trade shows (London, Birminhgam) and 2 conferences (Stuttgart, Helsinki) on behalf of the company and learnt a lot from it. Others on my team got sent to San Francisco for Photonics West, which is the largest photonics symposium, and ECCV, which is the largest computer vision event in Europe. It did help that I was a customer-facing engineer but these are some tips if you spot an event you want to attend.

The first thing you must do if you want to take on these opportunities, especially if you’re young, is to establish yourself as a competent, eloquent and trusted team member. Your line manager should feel comfortable to send you out externally knowing that you can represent the group/ company and make a good impression on others. It took around 6-9 months for me to get to this stage after doing some internal presentations, client calls and reports. After that, you should start noting down events you’d like to attend and start asking!

One great mentor at my previous company said the most stupid thing you can do is not asking!

Tips for asking your boss for sponsorship to attend conferences/ summer schools

🤘🏼In case you are also a young professional working in industry, I think joining conferences and summer schools are a good way to learn about things outside your day-to-day and to meet interesting people. SIGGRAPH have a really good “Get your boss on-board” template article which you can use to persuade whoever is in charge. I am not too familiar with the process in academia, probably there is a formal application for funding from specific funding bodies, but I am sure that there are similarities!

In summary you should mention

what the event is about? What is the field it is in? How big is it? Is is prestigious? Make sure it is of a decent size and actually relevant to your team/ company.
the value it brings to the company/ group (emphasise this)
1. Marketing value: is there a poster you can present to the audience? Remember to include an estimate of how many people attend the event and the split (industry/ academia)
2. Practical value: what methods can you learn from the conference and bring back to the team? Is there a talk that you would like to attend?
3. Networking value: will this generate new leads/ start new collaborations?
that it’s also for your personal development
Also remember to give an estimate of the costs and a detailed breakdown (transport, accomodation, registration, food)
When is the decision deadline?
That you will write a report so other team members can learn from your experience

What else can you do?

Some conferences have options to present industry insights/ posters/ conference papers. Even if you don’t work in a research team, some conference workshops have quite high acceptance rates so it’s always worth giving it a try.

Your line-manager is not obliged to give you this opportunity, but there’s nothing stopping you from putting the best case forward!

The Missing Semester- Tips for Scientists who code

2025-08-28T12:00:00+00:00

This article/ series inspired by the MIT Missing Semester course lectures, which I thought was oriented much more towards computer scientists and goes through the shell, build processes, remote access etc. These topics are important, but I thought there should be something more oriented towards the types of work scientists like myself do, which largely comprise of signal/ data processing, analysis and its automation. I started making some reels on my own ‘Missing Semester’ and this is a longer written form of the same information.

Most scientists now code in the Python programming language, which is a high level language that is now taught in many universities and practically standard in industry.

Unlike physics/ mathematics, a lot of these are just tools that us humans have designed/ built to make things easier/ more convenient.

There’s nothing fundamental about any specific programming language or a packager, and these guidelines and ‘best’ practices may be superceded in the future. It’s more important to understand WHY we do this, say for reproducibility, modularity, separation of concerns, hardware independence etc.

We will move from setting up your environments to using AI for coding assistance

1. Environments and Conda

Have you ever wondered why there are so many different Pythons on your computer? Each is a different Python interpreter, which reads and runs your python file, line by line. Each interpreter has its own environment, which gives access to libraries (code that other people have written).

To find out which python interpreter you are using, you can do which python in your terminal.

As we want analysis work to be reproducible and reusable, it is important to specify what libraries and packages the script requires, and also their versions. These are also called dependencies.

I use conda, which is a package manager. It’s super beginner friendly and you’ll see many repositories using this to specify dependencies using an environment.yml file. You then do conda env create --file environment.yml to create an environment using that file.

The other way usually taught is to use virtualenv with a requirements.txt file, but it doesn’t manage version conflicts of packages.

In short, you give recipe files to environment managers and specify what versions of Python and libraries your code should be run in so others can reproduce your findings.

2. Code organisation: Jupyter vs Python

I love Jupyter notebooks for quick prototyping—plots stick around, and it’s super fast to experiment with. There’s even VSCode extension for it!

But… analysis in notebooks is tough to reproduce, as code cells can be run in any order.

So here’s my approach: whenever I create a reusable function, I refactor it into a Python package and install it in editable mode.

Refactoring just means making the code flow better by shifting the code to another file/ restructuring it. These new modules can then be imported nicely inside another analysis notebook, following the ‘don’t repeat yourself’ (DRY) principle :)

Say I’ve copy and pasted my code into My code is now

my_repository
|_ my_package
    |_ my_file.py
    |_ __init__.py
|_ notebooks
    |_ data_analysis.ipynb
|_ pyproject.toml
|_ setup.py

You also need to create and a pyproject.toml.

# pyproject.toml
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

and a setup.py

# setup.py
from setuptools import find_packages, setup

setup(
    name="my_package",
    version="0.1.0",
    packages=find_packages(),
    install_requires=[],  # Add dependencies here
    author="",
    author_email="",
    description="A short description of your package",
    url="https://github.com/yourusername/your_package",
    classifiers=[
        "Programming Language :: Python :: 3",
        "Operating System :: OS Independent",
    ],
    python_requires=">=3.7",
)

I can now install my module in editable mode (-e), which means that any code I change in my module will instantly have effect, and there’s no need to reinstall pip install -e name_of_module.

In my Jupyter notebook (data_analysis.ipynb), I also put these two lines at the top

%load_ext autoreload
%autoreload 2

which ensures that any changes to my packages are instantly reflected in my Jupyter environment.

In this way, I keep prototyping separate from real development—making my code modular, reproducible, and reusable.

More info here and here for those who want deeper dives.

3. Git vs GitHub

Don’t make this common mistake—Git and GitHub are not the same thing. So, what’s Git?

Git is the standard version control system for programming projects. It tracks every line of change, lets you roll back if needed, and makes it possible for people to collaborate.

Version control systems track history

Think of a repository as your project, and a commit as a snapshot update. You can branch off, experiment, and later merge changes back into the main branch. And here’s the cool part—you don’t even need the internet. Git lives in a hidden .git folder on your computer.

Now, GitHub is different. It’s an online service built on top of Git—like Google Drive for code. It’s where you upload repositories to share, collaborate, and contribute with others. GitHub is just one of many platforms—others include GitLab and Bitbucket.

If you’re working with yourself, I’d recommend setting up your (remote) project on GitHub and then cloning it on your local PC.

Don’t be intimidated by the terminal

What is a Programming language?

Atomic advice for young me

2025-06-04T12:00:00+00:00

Thoughts

These are some notes I have shown to me every morning when I wake up, hopefully they resonate with you.

Treat people around you well and spend quality time with friends and family, because they will lift you up.
Believe in yourself, accept and love yourself and face yur imperfections. If you don’t, who will?
Live in the present, do what makes you feel alive and live intensely. Really really want things, but chill out cause everything it going to be okay.
Always produce, work really hard and lead your life well. Put yourself out there and good things will happen.
Don’t think that the world is efficient, nor perfect in its current state. There are always things to discover, problems to be solved and wrongs waited to be righted.
Everyone wants connection. Make small talk with people and listen to their story. Learn about people. We’re all on the same boat.
Protect your time. Hell yeah or no.
Be kind to others, without being asked to, and then good things will happen
Adopt skepticism, be curious, think if things are true. Form a strong view of the world and hope to be contradicted by others.

Quotes

“You don’t need to be great to get started, you just need to get started to be great.” 千里之行，始於足下。
“The only way to do great work is to love what you do. If you haven’t found it yet, keep looking. Don’t settle.” Steve Jobs.
“Let your hopes, not your hurts, shape your future.” Robert H Schuller
“Define success on your own terms, achieve it by your own rules, and build a life you’re proud to live.” Anne Sweeney
If you look at what you have in life, you always have more. If you look at what you don’t have in life, you’ll never have enough. This echoes 知足者常樂. Oprah Winfrey.
The privilege of a lifetime is being who you are. Joseph Campbell
It’s all about the people
Be the change you wish to see in the world
You will eventually use everything you’ve ever learned.
Go where you’re celebrated, not tolerated
Comparison is the thief of joy
If you cannot find peace within yourself, you will never find it anywhere else
We don’t read and write poetry because it’s cute. We read and write poetry because we are members of the human race. And the human race is filled with passion. And medicine, law, business, engineering, these are noble pursuits and necessary to sustain life. But poetry, beauty, romance, love, these are what we stay alive for.
Nothing is so fatiguing as the eternal hanging on of an uncompleted task.

My Research interests

2025-06-04T12:00:00+00:00

This is my attempt to understand some gaps in research that I currently find interesting in. At the moment they revolve around computer vision and machine learning in biology. Last updated: June 2025.

Computer Vision: World models and 3D

There has traditionally been a gap between the geometric reconstruction based side of computer vision with recognition/ reorganisation side which has shifted to neural network based. Transformer based architectures are now used to tackle the geometry based problem (VGGT, 2025), which is orders of magnitude quicker than other techniques such as structure from motion. Now this is present, it would be interested to tap into its embeddings to see what ‘understanding’ of the visual world it has.

Other topics: Neural Radiance Fields, 3/4D Gaussian Splatting

Groups: Visual Geometry Group, Oxford

Computer Vision: For science

A large part of the computer vision community is focused on visual perception and reasoning.

Cell painting (2016) is an imaging-based cellular assay based on morphology (size, shape) that can be used to quantify the effects of chemical or genetic ‘perturbations’ to the cells.

Image processing/ analysis techniques are maturing now (at least in 2D), and research is focused on obtaining an understanding of the objects in images themselves.

Computational Photography and Imaging

the joint design of optics and algorithms to get the result you want. Applications in cell microscopy and neuroscience. See Laura Waller, Na Ji.

Machine Learning for biology and medicine

Human scientists can pose questions/ hypotheses, do experiments, get data and learn from the data. Machines and computers are getting to the point where it can do the same. Centres such as the CZ Biohub, and big pharma are are constructing high throughput profiling platforms for basic research as well as applied screening.

Biological data is tricky as unlike internet images, many times the datasets are small, require expert annotation, and experiments are costly to run.

The world is inherently multimodal. Part of this is learning a ‘joint representation’ and hence developing understanding of the phenomenon itself.

Generative design of molecules. Graph neural networks lend naturally to molecules. How can it be applied in synthetic biology and therapeutics?

Meetings

Recent musings– podcasts and videos

2025-04-18T12:00:00+00:00

Recent podcasts I’ve been listening to

Machine Learning

It’s really interesting learning about the lives of researchers too and also the process of scientific research itself

The Robot Brains Podcast by Pieter Abbeel
- Biographic interviews on the greatest researchers in machine learning
- Host is an expert on reinforcement learning and robotics
- Favourite episodes: Fei-Fei Li, Andrej Kaparthy, Ilya Sutskyaver, Yann LeCunn
DeepMind Podcast hosted by Hannah Fry
- Episodes about DeepMind’s research into artificial intelligence
- Favourite episodes: Jeff Dean, Pushmeet Kohli, AI for science

Computational Imaging

Scien Seminars

Don’t be intimidated. Starting again in ML.

2025-04-18T12:00:00+00:00

Learnings

The first step is to not be intimidated
There’s an abundance of resources (papers, blog posts, frameworks) online and sometimes you just want a single source of truth
1. Choose uni lecture courses to see if your level matches (Stanford, Cambridge)
2. The heart of DL is mathematics, but you need to be able to build it with computational tools. Try to get to the bottom of it
3. Textbooks are good, until they go too deep
“What I cannot create, I do not understand”– Richard Feynman.
1. Define a simple task, get the base dataset, and do it.
2. Focus on understanding the concept and then implementing a working version of it
Find infra tools that help you (weights and biases)
ALWAYS validate on something that a human can intuitively understand. The loss, accuracy metrics can be deceptive
Talking through with others help a lot

Resources and inspirations

Andrej Kaparthy’s Recipe for Training Neural Networks
- Become one with the data
- Set up the end-to-end training/ evaluation skeleton + visualise EVERYTHING + get dumb baselines. Even better if you deploy it (especially if you want to show people)
- Overfit on the training set
- Last step: Regularise, tune, ensemble

Backpropagation of the batch norm layer

2025-04-01T12:00:00+00:00

Assignment 2 of Stanford’s CS231n took me a while to wrap my head round. These are some notes so that I remember.

Backpropagation

A gradient estimation method using reverse mode differentiation!

In learning algorithms, the gradient we most often require is the gradient of the cost function with respect to the parameters, $\nabla_W L(W)$ where $W$ are the weights and $L$ is the scalar loss function.

Batch Norm

Algorithm from the original paper

Forward pass

# Elementwise statistics, average over batch
sample_mean = np.mean(x, axis=0)
sample_var = np.var(x, axis=0)
denom = np.sqrt(sample_var + eps)
# normalise
x_corrected = (x - sample_mean)/ denom #xhat
out = gamma * x_corrected + beta #yi

# Track the statistics of the mean and variance for test time
running_mean = momentum * running_mean + (1 - momentum) * sample_mean
running_var = momentum * running_var + (1 - momentum) * sample_var

cache = {'x_hat':x_corrected,
'mu':sample_mean,
'var':sample_var,
'denom':denom, 'gamma':gamma, 'beta':beta, 'x':x
}

Method 1: Computational Graph

We trace the computational graph backwards step by step. This was explained here quite well!

dgammax = dout # dL/d(gamma*x)
dgamma = np.sum(dgammax * xhat, axis=0) #dL/dgamma
dxhat = dgammax * gamma # dL/dxhat
dbeta = np.sum(dout, axis=0) # dL/dbeta

dnom_1 = dxhat * (1/denom) #dL/d(x-mu)
ddenom = np.sum(dxhat * - nom/ denom**2, axis=0) # dL/d(sqrt(var+eps))
dvar = ddenom * 0.5/denom #dL/dvar

dnom_2 = dvar * 2 * x / N #dL/d(x-mu)
dnom = dnom_1 + dnom_2 #dL/dnom
dmu = -1 *  np.sum(dnom, axis=0) #dL/dmu
dx_1 = dmu * np.ones((N,D)) / N
dx_2 = dnom
dx = dx_1 + dx_2

Method 2: Differentiation from first principles

This is already well explained in this blogpost, but I’d just like to add some more intuition.

Example 1: Say we have a zero-centering/ mean subtraction operation using statistics derived from a batch. So $y = x - \mu$. We know that $x$ and $y$ have the same size $x,y \in \mathbb{R}^{(N\times D)}$ where $N$ is the batch size and $D$ is the number of dimensions in your training example. If we write all the entries of matrix $y$ out this will be

\[y_{kl} = x_{kl}- \frac{1}{N}\sum_{i}^{n}x_{il}\]

Equation 1: mean shift operation.

more verbosely, using the Kronecker delta notation, which explicitly tells you the rules of the transition from $x$ to $y$, and we label the indices $i,j$ and $k,l$ differently for matrices $x$ and $y$ for clarity.

\[y_{kl} =x_{ij}\delta_{ik}\delta_{jl} - \frac{1}{N}\sum_{i}^{n}x_{ij}\delta_{jl}\]

Equation 1: matrix entry view of batch mean subtraction. This tells you how each element in the training batch will affect the output.

Say we already measured how much $y$ impacts the scalar loss $L$. In other words, we know the upstream derivative, a 2D matrix $\frac{dL}{dy_{kl}}$, which is how much the loss $L$ changes by as each entry in $y_{kl}$ changes. However, we want to use backpropagation to compute the loss with respect to $x$. We use the chain rule and sum over the indices in $y$.

\[\frac{dL}{dx_{ij}} = \sum_{kl}\frac{dL}{dy_{kl}} \frac{dy_{kl}}{dx_{ij}}\]

The complicated thing here is that $x$ and $y$ are matrices/ tensors. This is explained away in the deep learning book.

The terms

$\frac{dL}{dy_{kl}}$ as explained, a 2D matrix
$\frac{dy_{kl}}{dx_{ij}}$ is called a Jacobian matrix here (2x2 = 4D), which explains how much each element in matrix $y$ changes as each element in matrix $x$ changes.

We write down the Jacobian computing the derivative of Eq. 1

\[\frac{dy_{kl}}{d x_{ij}} = \delta_{ik}\delta_{jl}-\frac{1}{N}\delta_{jl}\]

Eq. 2: Jacobian of the mean shift operation

Now we compute the loss by subtituting Eq.2 into the sum

\[\frac{d L}{d x_{ij}} = \frac{dL}{d x_{ij}} - \frac{1}{N} \sum_k \frac{dL}{dy_{kj}}\]

Eq. 3 Backpropagated gradient wrt $x$ of the loss function.

See that $\delta_{jl}$ is in both terms, which means that the channels are taken to be independent in the mean shift, as expected :)

Some additional interpretation: Consider the first term in Eq. 1, as $x$ changes, the same amount is changed in $y$. The second term is a bit trickier. Changing one entry of $x$ affects the batch mean, and so affects the ‘zero point’ of the entire batch, affecting every entry in $y$! But by how much? In fact a change $dx$ in $x$ will result in the outcome of each entry of $y$ to change by $dy = 1/N$. However, we ultimately care about the effect on the loss, which is why we are summing over indices $kl$. The insight here is that the term $\mu$ means that each row of $\frac{dL}{dx_{ij}}$ will have contributed to all $y_{kl}$.

Example 2: Say we have a variance operation. So $Y = \frac{1}{N}\sum_i(X - \mu)^2$.

\[\frac{\partial \sigma^2_l}{\partial x_{ij}} = \frac{2}{N} (x_{ij} - \mu_l)\delta_{jl}\]

Interpretation

each element $x_{ij}$ only affects the variance in the dimension that its relevant in, due to the Kronecker delta
a change in the mean itself will not affect the variance if the values’ excess from the mean stays the same (think of the variance being the same when the entries are all shifted by a constant amount)

Full Solution to Batch Norm:

\[\frac{\partial L}{\partial x_{ij}} = \sum _{kl} \frac{\partial L}{\partial y_{kl}}\frac{\partial y_{kl}}{\partial x_{ij}}\]

Use the product rule

\[\frac{\partial y_{kl}}{\partial x_{ij}} = \gamma_l \bigg [ \frac{\partial (x_{kl} - \mu_l)}{\partial x_{ij}}\cdot (\sigma^2_{l} + \epsilon)^{-\frac{1}{2}} + \frac{\partial (\sigma^2_{l} + \epsilon)^{-\frac{1}{2}} }{\partial x_{ij}}\cdot (x_{kl} - \mu_l) \bigg] \\ = \gamma_l \bigg[ (\delta_{ik}\delta_{jl} - \frac{1}{N}\delta_{jl}) \cdot(\sigma^2_{l} + \epsilon)^{-\frac{1}{2}} -\frac{1}{2} (\sigma^2_{l} + \epsilon)^{-\frac{3}{2}} \frac{2}{N} (x_{ij} - \mu_l)\delta_{jl}\cdot (x_{kl} - \mu_l) \bigg]\]

Now we sum over indices $k,l$ $\frac{\partial L}{\partial x_{ij}}$

In code, this is

N, D = dout.shape
gamma = cache['gamma']
denom = cache['denom'] # sqrt(var + eps)
mu = cache['mu']
var = cache['var']
x = cache['x']
nom = x-mu
xhat = cache['x_hat'] # (x-mu)/denom

dgamma = np.sum(dout * xhat, axis=0) #dL/dgamma
dbeta = np.sum(dout, axis=0) # dL/dbeta

sum_dout_batch = np.sum(dout, axis=0) #sum_k dL/dout_kj
sum_doutnom_batch = np.sum(dout * nom, axis=0) #sum_k dL/dout_kj * xkj
dx = (1 / N) * gamma * denom **(-1) * (N * dout - sum_dout_batch - nom * denom**(-2) * sum_doutnom_batch)

which leads to a 1.6x speedup roughly.

Side notes

Kronecker delta

Physicists here will be reminded of their classes in general relativity (tensors), atomic physics (dipole matrix transition elements).

Differences in resources

The deep Learning book draws nodes as intermediary variables

CS231n draws nodes as operations and treat nodes as ‘gates.’

The PyTorch docs illustrate both albeit in different shapes.

Additional notes on backprop

https://colah.github.io/posts/2015-08-Backprop/ (Good for understanding Computational graphs and why backprop matters)
Deep Learning book (Mathsy, as usual, good for a second pass once you’ve understood the material)

A Taylor expansion of effort

2024-11-29T12:00:00+00:00

I’ve recently been reading Greta Thunberg’s ‘The Climate Book,’ a collection of articles by climate experts and also Thunberg herself on the climate crisis and what we can do to help.

My existing view is that the world is heading on a a collision course towards disaster, the warming across the globe that leads to an increase in natural disasters, which will destroy homes and habitats for people and animals. It sometimes seems that what I can do, as an individual, is completely imperceptible, compared to the corporations and institutions that control the world, its energy supply, and the systems that people live in.

It’s like how our habits as a consumer, is controlled by what’s on discount, what’s shown to us on ads, and what the norm is.

What makes this book powerful is that it acknowledges this fact, that individual action is miniscule, but states that some individuals tend to have more power than others in shaping the future.

As a person living the global North, who gets on a plane for quite a few times each year, my ‘carbon footprint’ is already disproportionately larger than billions of people around the world. This means that my choices as a consumer already have a markedly bigger impact than maybe 90% of the world’s population.

Above that, as a physicist, I can contribute to the technology needed to produce renewables, capture carbon or replace materials with high carbon emissions.

As a person with a voice who’s not living in a cave, I can influence the people around me to adopt better practices.

And if enough people do this, there’s bound to be a person sufficiently connected to me in a position of power, whose actions can truly shape the environment.

If we expand a person’s (positive) impact $I$ as a function of effort $E$, it’s clear that consumer actions lead to a linear impact. However, building useful technologies, doing useful research, adding to the climate discussion, contributing to education and advocacy, can have lead to superlinear effects.

\[\text{Impact} = \frac{dI}{dE}\Delta E + \frac{d^2I}{dE^2}(\Delta E)^2 + \frac{d^3I}{dE^3}(\Delta E)^3 + \text{higher order terms}\]

This perspective was empowering to hear.