It reminds me of the latest, most advanced steam locomotives from the beginning of the 20th century.
They become extremely complex and sophisticated machines to squeeze a few more percent of efficiency compared to earlier models. Then diesel, and eventually electric locomotive arrived, much better and also much simpler than those late steam monsters.
I feel like that's where we are with LLM: extremely smart engineering to marginally improve quality, while increasing cost and complexity greatly. At some point we'll need a different approach if we want a world-shattering release.
Even more connections, the UPs Big Boy was designed to replace double header or helper engines to reduce labor costs because they were using the cheapest bismuth coal.
It wasn't until after WWII when coal and labor costs and the development of good diesel electric that things changed.
The UP expended huge amounts of money to daylight tunnels and replacing bridges in a single year to support the big boys.
Just like AI it was meant to replace workers to improve investors returns.
Just to add to this, if we see F1 cars a way to measure the cutting edge of cars being developed, we can see cars haven't become insanely faster than they were 10-20 years ago, just more "efficient",reliable and definitely safer along with quirks like DRS. Of course shaving off a second or two from laptimes is notable, but not an insane delta like say if you compared a car from post 2000s GP to 1950s GP.
I feel after a while we will have specialized LLMs great for one particular task down the line as well, cut off updates, 0.something better than the SOTA on some benchmark and as compute gets better, cheaper to run at scale.
To be fair, the speed of F1 cars is mostly limited by regulations that are meant to make the sport more competitive and entertaining. With fewer restrictions on engines and aerodynamics we could have had much faster cars within a year.
But even setting safety issues aside, the insane aero wash would make it nearly impossible to follow another car, let alone overtake it, hence the restrictions and the big "rule resets" every few years that slow down the cars, compensating for all of the tricks the teams have found over that time.
(I agree with the general thoughts on the state of LLMs though, just a bit too much into open-wheel cars going vroom vroom in circles for two hours at a time)
> if we see F1 cars a way to measure the cutting edge of cars being developed
As other commenters noted, F1 regulations are made to make the racing competitive and interesting to watch. But you can design car that would be much faster [1] and even undrivable for humans due to large Gs.
They become extremely complex and sophisticated machines to squeeze a few more percent of efficiency compared to earlier models. Then diesel, and eventually electric locomotive arrived, much better and also much simpler than those late steam monsters.
I feel like that's where we are with LLM: extremely smart engineering to marginally improve quality, while increasing cost and complexity greatly. At some point we'll need a different approach if we want a world-shattering release.