2 minute read

This is my attempt to understand some gaps in research that I currently find interesting in. At the moment they revolve around computer vision and machine learning in biology. Last updated: April 2025.

Computer Vision

World models and 3D

There has traditionally been a gap between the geometric reconstruction based side of computer vision with recognition/ reorganisation side which has shifted to neural network based. Transformer based architectures are now used to tackle the geometry based problem (VGGT, 2025), which is orders of magnitude quicker than other techniques such as structure from motion. Now this is present, it would be interested to tap into its embeddings to see what ‘understanding’ of the visual world it has.

Other topics: Neural Radiance Fields, 3/4D Gaussian Splatting

Groups: Visual Geometry Group, Oxford

For science

A large part of the computer vision community is focused on visual perception and reasoning.

Cell painting (2016) is an imaging-based cellular assay based on morphology (size, shape) that can be used to quantify the effects of chemical or genetic ‘perturbations’ to the cells.

Image processing/ analysis techniques are maturing now (at least in 2D), and research is focused on obtaining an understanding of the objects in images themselves.

Machine Learning for biology and medicine

Human scientists can pose questions/ hypotheses, do experiments, get data and learn from the data. Machines and computers are getting to the point where it can do the same. Centres such as the CZ Biohub, and big pharma are are constructing high throughput profiling platforms for basic research as well as applied screening.

Biological data is tricky as unlike internet images, many times the datasets are small, require expert annotation, and experiments are costly to run.

The world is inherently multimodal. Part of this is learning a ‘joint representation’ and hence developing understanding of the phenomenon itself.

Generative design of molecules. Graph neural networks lend naturally to molecules. How can it be applied in synthetic biology and therapeutics?

People:

  • Maria Brbic
  • Marika Zitnik
  • Jure Leskovec
  • Mahmood lab: AI for Pathology

Meetings

Other interests (past and present)

  • computational photography: how do you best process your images? See Paul Debevec.
  • computational biophotonics: the joint design of optics and algorithms to get the result you want. Applications in cell microscopy and neuroscience. See Laura Waller, Na Ji.
  • the information content in hyperspectral imaging.
  • medical imaging: see Daniel Rueckert

Updated: