Instead of time it should be energy. What is the best model you can train with a given budget in Joules. Then the MBP and the H100 are on a more even footing.
Also, my laptop running Linux and its outputs are probably mine and private. If I use cloud GPU's, I need to be a lawyer to be sure what they can or can't do with my data or models.
There's also no overages or hidden charges with a laptop. Past simply breaking it. You know the replacement cost ahead of time, though.
H100s are almost-instantly available to anyone with a credit card and access to the internet. Without even having to lift their butt from the seat. And you get plenty more than five minutes of compute for the price of an M4.
While I love cloud computing, you're comparing the cost of renting a GPU for a fixed amount of time to the purchase of an asset which can be used for years.
Not a useful comparison IMHO.
Disagree, equity of access matters a lot. Not everyone benefits from exposure to the entire hardware lifecycle, the same way that buying housing is not the best financial decision for everyone regardless of affordability. I might have unlimited budget but if I only need access to state of the art hardware intermittently or under irregular circumstances the cost of renting may be efficient for my needs. Also consider the costs of supporting hardware that is fully owned, if you own the hardware but underutilize it that is inefficiency and the owner bears that cost. The unusual way that silicon depreciates mean that the value of your “asset” is not static and rapidly depreciates as silicon manufacturing improves.
For the orgs where I've worked the important thing isn't availability of compute it's security. Using what we have on our local network is much easier from a governance and approval standpoint than whatever is available on the internet.
Many orgs have no problems using cloud envs for most things. The usual suspects offer just as secure compute envs as everything else.
Anyway, I was assuming personal use, like the messing-around experimenting that the article is about. (Or who knows, maybe it was part of the author’s job.)
And yet just about any intro-to-programming tutorial gets something running on your local machine, and local machine development continues to be the default for most people, even though devving on a cloud machine is eminently reasonable.
"Pull out credit card, sign up for some thing and pay a bit of money" is a non-trivial bit of friction! Extremely non-trivial!
Especially in a corporate context - you have to get the expense approved. It's not clear if you can put company data onto the machine. Whereas generally running local things on corporate laptops is far less controversial.
"Download this tool and run it." is still an extremely powerful pitch. Pretty much the only thing that beats it is "go to this website which you can use without any signup or payment".
Yeah, is a large server rack to run those H100s. But realistically, the majority of people have a PC with consumer grade GPU or more likely a laptop with...laptop grade GPU.
Cloud H100 don't count because you need lawyer to review ToS and other agreements.
Frankly I think a lot of full-time-employed technical people are largely experimenting for fun in the context of things that might eventually be useful to their employer. AI is cool and fascinating stuff and when I have a few idle minutes at the end of my workweek I love catching up and experimenting with the latest and greatest, but with an eye towards company problems and on company time, and sometimes using company datasets. That means company vendor approval and financing of my efforts.
In my personal life, when its time for fun, I close the laptop and go do some gardening.
Maybe not to buy one, but to rent one. Like how barista-made coffee is an everyday product even though most people can't afford a fancy professional coffee machine.
Reasonably high quality coffee machines are very widespread. Or you can do pour-over. I don’t think the cost of a machine is a limiting factor for many people, it is just convenience.
Maybe an analogy could be made to espresso, nice espresso machines get costlier. But, you can still get quite good results out of a manual machine like a Flair.
I think this is why the suggestion to rent a machine is not to helpful. In this analogy we’re on BaristaNews, we all know about the industrial machines, lots of folks use them at work. But, the topic of what sort of things you can do on your manual machine at home has come up.
> Reasonably high quality coffee machines are very widespread. Or you can do pour-over. I don’t think the cost of a machine is a limiting factor for many people
No, reasonably-priced coffee machines is an enabling factor for many people.
If coffee machines weren't reasonably priced, they would not be "very widespread".
Mac is more competitive on power consumption though since its not ever pulling as much as a Nvidia GPU is my understanding.
On that note you can rent an H100 for an hour for under $10 which might make for a slightly more interesting test, whats the best model outcome you can train in under an hour.
It depends. If you're bottlenecked by memeory speed, the Mac typically comes out on-top.
In terms of conpute efficiency though, Nvidia still has Apple beat. Nvidia wouldn't have the datacenter market on a leash if Apple was putting up a real fight.
> Instead of time it should be energy (...) Then the MBP and H100 are on a more even footing.
What exactly is your point? That instead of expressing workloads in terms of what a laptop could do, you prefer to express them in terms of what a MacBook Pro could do?
The point is that "best model you can train in 5 minutes" is hardware dependent, the answer will be different depending on the hardware available. So it's necessarily a single-player game.
"Best model you can train with X joules" is a fairer contest that multiple people could take part in even if they have different hardware available. It's not completely fair, but it's fair enough to be interesting.
Training models with an energy limit is an interesting constraint that might lead to advances. Currently LLMs implement online learning by having increasingly large contexts that we then jam "memories" into. So there is a strict demarcation between information learned during pre-training and during use. New more efficient approaches to training could perhaps inform new approaches to memory that are less heterogenous.
