This is a very good primer, and it’s now time to think about second-order effects.
1. Current HBM designs —especially the CPU-HBM stacks you describe—are massive heat producers, so much so that they require new approaches to heat dissipation. That’s another constraint on production.
2. Given that reality, it’s not unreasonable to consider heat as another “product” of AI compute. I’m not a biochemist, but it seems rather obvious that the industry that could use both products is pharmacological R&D, especially for novel protein synthesis.
3. Power consumption is another area of intense effort in chip design. Every major fabricator is working to increase efficiency in model usage. We will eventually discover ways to get better inference from smaller, more constrained models. That will reshuffle the thinking about memory. Finally;
4. Anthropic’s new Mythos model has uncovered a massive new market—home AI appliances. The attack surface area for cybercrime now includes all home routers and so-called smart devices. The only reasonable defense is likely to be a home AI sentinel that incorporates threat detection and patching into a home router.
Taken together—and this is a quick list off the top of my head—we’re looking at transformational forces coming for energy usage and distributed computing.
The HBM concentration point is underappreciated. The market tends to price the end-chip designer (NVDA) and discount the memory vendor as a commodity supplier, but the physics of training at H100/B200 scale makes HBM a genuine bottleneck that doesn't get substituted away quickly.
Heat is definitely what I am considering. I did a first pass article on this and it's why I am more focused on the move to optical/plasmonics products in the computing stack. But it's a ways off (if ever) until the entire stack gets supplanted and the integration back to copper will always present new challenges and bottlenecks.
Great summary of how memory changed with HBM and why this trend deserves a higher valuation multiple.
Thank you brother
Great post! Nvidia is forming a monopoly in the AI space.
This is a very good primer, and it’s now time to think about second-order effects.
1. Current HBM designs —especially the CPU-HBM stacks you describe—are massive heat producers, so much so that they require new approaches to heat dissipation. That’s another constraint on production.
2. Given that reality, it’s not unreasonable to consider heat as another “product” of AI compute. I’m not a biochemist, but it seems rather obvious that the industry that could use both products is pharmacological R&D, especially for novel protein synthesis.
3. Power consumption is another area of intense effort in chip design. Every major fabricator is working to increase efficiency in model usage. We will eventually discover ways to get better inference from smaller, more constrained models. That will reshuffle the thinking about memory. Finally;
4. Anthropic’s new Mythos model has uncovered a massive new market—home AI appliances. The attack surface area for cybercrime now includes all home routers and so-called smart devices. The only reasonable defense is likely to be a home AI sentinel that incorporates threat detection and patching into a home router.
Taken together—and this is a quick list off the top of my head—we’re looking at transformational forces coming for energy usage and distributed computing.
Fun times.
Love that Steve. A lot to think about. I do wonder about new forms of compute that aren’t as heat producing.
Very interesting!!
The HBM concentration point is underappreciated. The market tends to price the end-chip designer (NVDA) and discount the memory vendor as a commodity supplier, but the physics of training at H100/B200 scale makes HBM a genuine bottleneck that doesn't get substituted away quickly.
Thank you
Great writing btw!
Heat is definitely what I am considering. I did a first pass article on this and it's why I am more focused on the move to optical/plasmonics products in the computing stack. But it's a ways off (if ever) until the entire stack gets supplanted and the integration back to copper will always present new challenges and bottlenecks.