Dancing with AI

What’s most striking about Palo Alto–based SambaNova is its ambition. Liang’s co-founders, Kunle Olukotun and Christopher Ré, are Stanford professors who in the mid-2010s teamed up to rethink computer architecture for an AI world. Ré was focused on machine learning and databases. Olukotun, who is best known for pioneering the multicore processor, a technology that led to major improvements in chip efficiency and speed and became ubiquitous in the early 2000s, focused on chips and systems. Rather than tweaking existing technologies to run ever-expanding neural nets, Olukotun and Ré decided they would start from scratch, imagining a new type of system built expressly for AI that would avoid bottlenecks and limitations of legacy chips and servers. Their solution was a new, full-stack computing architecture—from silicon to software—tailor-made for data-heavy AI algorithms. “Traditional chips, even those commonly used for AI applications, are not good at handling those data-heavy algorithms,” says Olukotun. “So we set out to design a computer that natively executes this paradigm of computing, which is core to machine learning models.”

Olukotun began talking with Liang in early 2017 about what he sometimes dubs “data flow computing” and sometimes “software 2.0.” The two knew one another from Afara Websystems, a chip startup Olukotun had founded in 2000 to capitalize on his idea for a multibrained chip built to keep pace with web traffic and the demands of heavily used servers. Olukotun had hired Liang to oversee not just the production of the novel chips but also the development and the build-out of a complete system. “Kunle and I see the world in a very similar way,” says Liang, who has worked on high-end computers since his first job at Hewlett-Packard in 1993. “You can’t just build these individual components without having the stack.” At Afara, the team’s ambitions collided with the dot-com crash of the early 2000s, when money for startups dried up. In 2002, Sun Microsystems bought the company for $28 million. When Olukotun laid out his idea for a new, AI-native system designed expressly for data-intensive computing, Liang jumped at the opportunity. “Kunle is a brilliant mind when it comes to systems,” Liang says. “He’s usually many, many years ahead of everybody else in what’s coming next.” Liang took on the role of CEO. As the company’s chief technologist and chief scientific advisor respectively, Olukotun and Ré split time between teaching at Stanford and the company. Liang, who was born in Taipei but grew up in Brazil, chose the name SambaNova, which means “new dance” in Portuguese. “It’s a new dance,” Liang says. “It’s time for us to do something new in the world of technology.”

SambaNova’s pioneering technology has had a proven impact with many high-profile customers. One early adopter, the U.S. Argonne National Laboratory, uses SambaNova’s systems for a variety of applications that range from improving weather forecasting to better predictions for how tumors respond to different drug combinations. The national lab has reported a tripling in performance speed. Says Vijay Tatkar, a director at SambaNova: “We’ve been able to train large language models six times faster than a Nvidia A100.” Other operations, he adds, have seen a 10x improvement in speed. This past September, SambaNova introduced a new chip, the SN40L, that wowed markets and the press, and vastly improved on the performance of its earlier models. The chip is designed to run AI models that are more than twice the size of the advanced version of OpenAI's ChatGPT and is tailor-made for LLMs in the enterprise, enabling companies to stand up their own ChatGPT equivalent in a matter of days. Its release, which coincided with a chip shortage that pushed customers to find new suppliers, further positioned the company as a direct competitor to Nvidia in the market for AI computing.

Prior to November 30, 2022, when SambaNova’s people knocked on the doors of the large enterprises they identified as the natural customers for its high-end systems, they were met by worries about an economic slowdown and skepticism around AI. Company executives wished for tailwinds that would help boost sales. Instead, they were met by hurricane forces. “Before November, almost every conversation with a customer started with us trying to convince them of AI’s usefulness,” says Sumti Jairath, a member of the company’s founding team and its chief architect. “Now our starting point of 100 percent of the conversations is the concrete ways they might use AI.” Potential customers who have toyed with ChatGPT and played with image-making diffusion models “intuitively understand AI’s potential,” Jairath says. “Now they want to know how it can help their enterprise solve business problems.” SambaNova had its successes prior to the release of ChatGPT. An impressive roster of government and business customers had bought or leased its DataScale product, a refrigerator-sized server rack loaded with 32 of its chips, all sharing the same power supply, cooling system, and networking infrastructure. In 2021, SoftBank Vision Fund 2 led a $676 million funding round, making SambaNova temporarily the world’s best funded AI startup, with a valuation of more than $5 billion. “Rodrigo always said if we ever did another hardware company after Afara, he would make sure that we did not run out of money,” Olukotun says. Two months prior to OpenAI’s unveiling of ChatGPT, SambaNova released the third generation of its signature chip. “They’ve developed really innovative hardware and software solutions that dramatically accelerate workloads and make them really attractive in all sorts of very different industries, from autonomous vehicles to financial services to oil and gas exploration,” says Amit Lubovsky, an investment director at SoftBank Investment Advisers and a SambaNova board member.

Timetables were upended after November 30. One- and three-year projects were rendered obsolete. Prior to the launch of ChatGPT, it took months to close a deal. Since then, the excitement and urgency around generative AI has made the process much faster. “The velocity of the business has been supercharged,” Choy says. SambaNova became more focused. Previously, the company would stress that its chips were multipurpose. Its machines do seismic analysis for oil and gas companies, process satellite imaging for the U.S. government, and run scientific models on behalf of research institutions. But now all anybody wants to hear about is generative AI, and more specifically, LLMs, the technology behind ChatGPT and other conversational AI interfaces. “When November hit and we saw all the fervor around natural language processing, we leaned hard into the language side,” Liang says. “It was like, ‘Let us be your private GPT vendor.” The new SN40L chip bolsters that pitch even further. The interest in LLMs also caused SambaNova to reorient resources to focus on inference, the process of reasoning and making decisions based on available data. The company also launched a pretrained multilingual language model, BLOOMChat, for prospective customers to test the waters and fine-tune models with their own data. And it created a program that allowed startups to onboard onto the SambaNova platform and use it to develop new initiatives. “It’s a good thing to do for the entrepreneurial community and a smart way for SambaNova to stay on top of what’s going on in the market,” Lubovsky says.

While the ChatGPT release was a watershed moment for the company, it did not change the trajectory of the system’s design or its conviction for how companies should incorporate AI. SambaNova had bet on the relevance of open-source models for a long time. That conviction only solidified as SambaNova worked to ensure open-source models ran as efficiently as possible on its systems. “Our focus was engaging with customers around the idea of using open models and training those models on their own data to provide the capabilities that they needed within their organizations,” Olukotun says. There were challenges. Expedited sales are great for the bottom line, but they’ve also taxed SambaNova’s supply chain. “We have a lot more work [to do] trying to accelerate material to get things out to customers faster,” Liang says. “We need to do a lot more work on software releases. We need more coverage with staffing that supports our customers.” SambaNova has also had to contend with the backlash sparked by media hype around generative AI. The same outlets that wrote so enthusiastically about it have since become focused on its limitations. “Just three or four months after it started, people were talking about the cracks,” Liang says. “It was like, ‘Oh, it’s not secure, it hallucinates, it’s not auditable.’” But those negatives became an advertisement, of sorts, for SambaNova’s open-source philosophy. Nearly all the LLMs on the market are based on a famous 2017 paper from researchers at Google, and all use the same general architecture. What differentiates most is the nature and quality of the data used to train a neural network.

“We tell companies, ‘You want GPT? We’ll give you your own that’s secure, private, and trained on your own data,’ ” Liang says. The company pretrains open-source models on behalf of its customers, depending on what industry they’re in. Auditing, accuracy, and alignment are more straightforward when a customer knows exactly what data is used to train a neural network. Importantly, SambaNova, unlike some of its competitors, encourages customers to retain control of their own data and models, protecting critical IP, a model that “really resonates with businesses,” Liang says. “We're in the business of building people's models as assets. As soon as we've trained these models for customers using their data, 100% of the customers want to own that model.” As interest in SambaNova’s technology expands geometrically, the company is working hard to avoid being pulled in directions that could be counterproductive. “In a market like this one that’s brand new and offers a breadth of opportunities, we’ve made sure that a level of discipline permeates the company,” Liang says. One requirement: spending more time with customers to ensure they’re reacting to the market as it exists and not as they imagine it. As a result, customer-facing employees find themselves on the road more often than not. “On the business side of things,” Choy says, “we are just spending inordinate amounts of time outside SambaNova and inside the walls of our customers and discussing, learning, and aligning with them.”