Inside the AI Accelerator Market: GPUs, Custom ASICs, and Who Wins Inference

AI accelerators used to mean one thing: an NVIDIA GPU. Today the field is crowded. Hyperscalers build their own chips, AMD has landed a multi-gigawatt customer, and startups built only for inference are signing deals with the largest AI labs. This guide sets out who builds what, where each kind of chip fits between training and inference, and why the GPU vs ASIC debate now centers on serving tokens rather than training models.

Why the accelerator map changed

For most of the deep-learning era, the AI accelerator market ran on one flywheel. NVIDIA GPUs plus the CUDA software ecosystem were the default for both training models and running them. That default still holds in many places. What has changed is the shape of demand. Frontier labs and cloud providers now run models for very large numbers of users around the clock. The cost of producing each token, and the power that production uses, has become a first-order business problem.

The vendors say so openly. Google called its seventh-generation TPU, Ironwood, “the first Google TPU for the age of inference” [3]. Microsoft introduced Maia 200 as “the AI accelerator built for inference,” built “to dramatically improve the economics of AI token generation” [6]. In its own words, OpenAI’s compute strategy is “to build a resilient portfolio that matches the right systems to the right workloads” [11]. The phrase right systems to the right workloads sums up the market in 2026: no single chip has to win everything, and buyers are actively mixing vendors.

The incumbent: NVIDIA’s scale

Any map of AI inference chips and training silicon starts with NVIDIA, because nobody else operates at its scale. In its fiscal 2027 second-quarter results, published August 26, 2026, NVIDIA reported total revenue of $96.2 billion, up 106% year over year. Data Center revenue was $89.0 billion, up 117% from a year earlier [1]. The company’s CFO commentary filed with the SEC breaks that down further. Hyperscale customers accounted for $48.7 billion. The category NVIDIA calls “AI Clouds, Industrial, & Enterprise” accounted for $40.3 billion and grew 138% year over year [2]. The commentary attributes the growth to “the ramp of our Blackwell Ultra infrastructure” [2].

NVIDIA’s announcements in the same release show how it is defending its position. It said its Vera Rubin platform is “ramping into full production,” with racks running at CoreWeave, Google Cloud, Microsoft Azure, Oracle Cloud Infrastructure and Nebius [1]. It also said “NVIDIA Groq 3 LPX, the interactive AI inference accelerator, is now in full production” [1]. That is notable. The world’s largest GPU vendor now ships a product line built on technology licensed from an inference ASIC startup (more on that below).

NVIDIA’s customer list also shows how labs hedge. In November 2025 the company announced a partnership with OpenAI to deploy “at least 10 gigawatts of NVIDIA systems.” It also said that “for the first time, Anthropic will run and scale on NVIDIA infrastructure, initially adopting 1 gigawatt of compute capacity” [12]. Anthropic is also one of AWS’s flagship Trainium customers [8][9]. Buying from several suppliers is now normal.

The other GPU: AMD’s multi-gigawatt foothold

AMD is the main merchant alternative in GPUs. Its big breakthrough came on October 6, 2025, when AMD and OpenAI announced “a 6 gigawatt agreement to power OpenAI’s next-generation AI infrastructure across multiple generations of AMD Instinct GPUs.” The first 1-gigawatt deployment of Instinct MI450 GPUs is set to begin in the second half of 2026 [4]. AMD describes the collaboration as one that “began with the MI300X and continued with the MI350X series.” It issued OpenAI a warrant for up to 160 million AMD shares that vests against deployment and share-price milestones [4]. AMD’s CFO said the partnership is “expected to deliver tens of billions of dollars in revenue for AMD” [4].

The strategic point is clear. A GPU buyer that wants leverage and supply security now has a second programmable-GPU source at frontier scale. For the GPU vs ASIC question, AMD sits on the GPU side. It sells flexible, general-purpose parallel processors to anyone who wants them, rather than chips designed around one company’s own workloads.

The hyperscaler ASICs

Custom application-specific integrated circuits (ASICs) trade generality for efficiency on a known set of workloads. Each of the four largest cloud and platform companies now has its own line.

Google TPU: from training workhorse to inference flagship

Google has the longest-running custom accelerator program. It says that “for more than a decade, TPUs have powered Google’s most demanding AI training and serving workloads” [3]. Ironwood is the seventh generation. Google’s Cloud documentation lists TPU7x (Ironwood) at 4,614 peak FP8 TFLOPs per chip, 192 GiB of HBM at 7,380 GBps per chip, and 9,216 chips per pod. That compares with 918 FP8 TFLOPs and 32 GiB for the previous-generation v6e (Trillium) [5]. Google also claims Ironwood’s performance per watt is 2x that of Trillium [3].

The details are worth noting. Google markets Ironwood as built for inference, but its documentation describes the generation as “designed for large-scale AI training and inference,” including “pre-training, sampling and decode-heavy inference” [5]. Google has also worked on portability. It says enhanced TPU support in vLLM lets developers “switch between GPUs and TPUs, or run both, with only a few minor configuration changes” [13]. Software lock-in has long been the GPU’s best defense, so lowering switching costs matters as much as raw FLOPs.

AWS Trainium (and the Inferentia lineage)

Amazon’s line is named for training, but AWS now sells it for both jobs. At re:Invent 2024, AWS said Trn2 instances “offer 30-40% better price performance than the current generation of GPU-based EC2 P5e and P5en instances” [7]. It also announced Project Rainier with Anthropic [7]. Amazon later said Project Rainier was “fully operational” with “nearly half a million Trainium2 chips.” It added that Claude was expected to run on “more than 1 million Trainium2 chips — for workloads including training and inference — by the end of the year” [8].

The next generation, Trainium3, is AWS’s first 3nm AI chip. Amazon says Trn3 UltraServers scale to 144 chips and deliver “up to 362 FP8 PFLOPs.” It claims up to 4.4x the compute performance and 4x the energy efficiency of Trainium2 UltraServers, and says “Amazon Bedrock is already serving production workloads on Trainium3” [9]. The name says training, but AWS clearly pitches the part for inference too.

Microsoft Maia 200: built for tokens

Microsoft’s Maia 200, announced January 26, 2026, is the clearest example of a hyperscaler chip built for inference. Microsoft says it is built on TSMC’s 3nm process with native FP8/FP4 tensor cores, “216GB HBM3e at 7 TB/s and 272MB of on-chip SRAM.” It delivers “over 10 petaFLOPS in 4-bit precision (FP4) and over 5 petaFLOPS of 8-bit (FP8) performance, all within a 750W SoC TDP envelope” [6]. Microsoft claims 30% better performance per dollar than the latest hardware in its fleet. It says Maia 200 will serve models including OpenAI’s GPT-5.2 for Microsoft Foundry and Microsoft 365 Copilot [6].

Two design choices show the inference focus. The first is the large on-die SRAM, which keeps data close to compute when decoding is limited by memory bandwidth. The second is the networking. Microsoft says Maia 200 uses “a novel, two-tier scale-up network design built on standard Ethernet,” with clusters of up to 6,144 accelerators, “without relying on proprietary fabrics” [6]. That puts it in direct contrast with NVLink-based GPU racks.

Meta MTIA: four generations aimed at serving

Meta’s MTIA program began with recommendation inference. In 2023, Meta wrote that “GPUs were not always optimal for running Meta’s specific recommendation workloads at the levels of efficiency required at our scale” [14]. In March 2026, Meta said it has “deployed hundreds of thousands of MTIA chips in production” and that the chips are developed “in close partnership with Broadcom” [10]. The roadmap shows where Meta thinks the bottleneck is. Meta says that “the bandwidth of high-bandwidth memory (HBM) is the most important factor affecting GenAI inference performance.” It doubled HBM bandwidth from MTIA 400 to MTIA 450, with mass deployment scheduled for early 2027, and added another 50% for MTIA 500, scheduled for 2027 [10]. Meta also stresses that it remains “committed to a diverse silicon portfolio,” meaning MTIA complements GPUs rather than replacing them [10].

The inference specialists: Groq and Cerebras

The startups show most clearly how valuable fast inference has become.

Groq designed its LPU architecture for low-latency inference. On December 24, 2025, Groq announced “a non-exclusive licensing agreement with Nvidia for Groq’s inference technology.” As part of the deal, founder Jonathan Ross, President Sunny Madra and other team members joined NVIDIA, while Groq continues as an independent company and GroqCloud keeps operating [15]. NVIDIA now lists an “NVIDIA Groq 3 LPX” inference accelerator in full production [1]. The incumbent chose to license an inference specialist’s technology rather than compete with it from scratch.

Cerebras builds wafer-scale processors. OpenAI announced on January 14, 2026 that it is partnering with Cerebras “to add 750MW of ultra low-latency AI compute to our platform,” with capacity coming online “in multiple tranches through 2028” [11]. OpenAI says Cerebras’ speed “comes from putting massive compute, memory, and bandwidth together on a single giant chip.” It presents the deal as adding “a dedicated low-latency inference solution” to its mix [11]. Press reports (Reuters, citing a Wall Street Journal report) put the deal at around $10 billion [16]; OpenAI’s announcement does not state a value [11].

GPU vs ASIC: training vs inference, side by side

A pattern stands out. Training at the frontier is still dominated by large, tightly coupled clusters. Merchant GPUs lead here, with Google’s TPUs and AWS’s Trainium as the main proprietary alternatives. Inference is where specialization is spreading. Microsoft and Meta built chips with inference as their main target. Google gave Ironwood an inference-first identity. Two startups with inference-only pitches won major deals in the space of a few weeks.

Market dynamics: what decides who wins inference

Rather than predicting a single winner, it is more useful to look at the levers the vendors themselves stress.

1. Memory bandwidth over peak FLOPs. Meta says this outright for generative AI inference [10]. Microsoft’s Maia design combines large on-die SRAM with HBM3e [6]. Cerebras’ pitch rests on putting memory and compute on one wafer [11]. Generating tokens one at a time is usually limited by how fast weights and KV-cache can be moved, so chips are now designed around data movement.

2. Performance per dollar and per watt. Almost every custom-silicon announcement leads with economics rather than raw speed. Examples include Microsoft’s claimed 30% performance-per-dollar gain [6], AWS’s claimed 30–40% better price performance for Trn2 against GPU instances [7], and Google’s claimed 2x performance per watt over Trillium [3]. These are vendor claims, not independent benchmarks. Still, they show what buyers are asking for.

3. Power as the unit of procurement. Deals are now sized in gigawatts and megawatts rather than chip counts: 10 GW for NVIDIA–OpenAI [12], 6 GW for AMD–OpenAI [4], 750 MW for Cerebras–OpenAI [11], and 1 GW for Anthropic’s initial NVIDIA commitment [12]. When available power is the binding constraint, efficiency per watt directly decides how much capacity a buyer can actually serve.

4. Software portability. CUDA’s moat is real, but the ASIC camp is going after it. Google promotes vLLM support across GPUs and TPUs [13]. Microsoft ships a Maia SDK with PyTorch integration and a Triton compiler for “easy model porting across heterogeneous hardware accelerators” [6]. Meta integrates MTIA directly into PyTorch [14]. The easier it is to move a model, the less it costs to choose an ASIC.

5. Scale-up networking. Large models are now served across racks, not single chips. NVIDIA sells integrated rack platforms such as Vera Rubin [1]. Microsoft is betting on Ethernet scale-up [6]. Google relies on its Inter-Chip Interconnect across 9,216-chip pods [3]. Meta’s MTIA 400 features “a 72-accelerator scale-up domain” [10]. The interconnect is increasingly part of the product.

So, who wins inference?

On the evidence available today, the honest answer is that inference is turning into a portfolio market rather than a single-winner one. NVIDIA remains by far the largest supplier by revenue [1], and it has moved to absorb inference-specialist technology through the Groq license [1][15]. Hyperscalers are moving more of their own serving onto in-house silicon, mainly where they control both the model and the platform: Microsoft with Copilot and Foundry [6], Meta with recommendations and GenAI [10], Google across its own services and Cloud [3], and AWS with Bedrock [9]. Frontier labs hedge across all of them. OpenAI alone has public agreements with NVIDIA, AMD and Cerebras [12][4][11], and Anthropic runs on both Trainium and NVIDIA [8][12].

For readers tracking AI accelerators, the practical lesson is to watch the per-token and per-watt economics that each vendor publishes. Read them as claims, compare them against independent benchmarks when they appear, and follow where the big labs place their next gigawatt. That is where the GPU vs ASIC contest will be decided.

Related reading

• Multi-Trillion-Parameter Training: What the Next Silicon Generation Must Deliver
• Capex Arms Race: Mapping Hyperscaler AI Spend and What It Buys
• InfiniBand vs Ethernet for AI Clusters: The Fabric War Inside the AI Factory

Images: illustrations generated for AndBeyond.Tech, except: NVIDIA H100 accelerator — 极客湾Geekerwan, Wikimedia Commons, CC BY 3.0; Google Tensor Processing Unit 3.0 board — Zinskauf, Wikimedia Commons, CC BY-SA 4.0.