What’s going on in ML?

It’s pretty simple.

Machine learning is just curve fitting.

The toy ML models that run on datasets like MNIST are just doing curve fitting.

The huge large language models ran at OpenAI & Google are just doing curve fitting.

ML is just curve fitting with a lot of parameters.

What’s a curve? What is fitting?

Well if you didn’t know, ML models need data to train. A lot of data.

You start my just visualizing the data. Like literally just graph it.

Here’s an example of a data points (blue circles):

Curve fitting is drawing a line or curve that best maps the data. The red is an example of a line. The green is an example of a curve.

Looking at the graph, the green (curve) better represents the data than the red line. It is closer to a lot more points.

This is all of machine learning. What is the line that can best fit my data? How can I make my line more like the green?

It’s kinda more complicated…

No, not more complicated in the sense that there is something other than curve fitting. It’s JUST curve fitting.

However, curve fitting with this many parameters, and maintaining generality is difficult.

How to get a curve to fit, while keeping it general (applicable to a large dataset), is the problem.

The black box

Another problem, which is weird for people outside of ML, is we don’t understand what these models are doing.

The properties of the model indicate that they are curve fitting, but how they are curve fitting, we don’t really understand.

There’s a lot of guesses on how these models work on the inside, but they are just guesses.

A whole field of ML is called Mechanistic Interpretability where researchers just try to uncover how these models learn exactly.

Continue reading

• My landing page

• Chris Olah

• Transformer Circuits

• Mechanistic Interpretability