CTO Survey: Generative AI is unlocking ROI in software development

CTOs have been on the frontlines of generative AI, guiding its adoption and deployment, making decisions about how to integrate it into products and operations. They’re also under pressure to recruit and develop AI talent within their companies — and to show their investments are paying off. Their answers to our detailed survey, two years into the generative AI era, paint a vivid picture of where the technology is making the biggest inroads, where it is providing value, and what challenges continue to hold it back.

Here’s a closer look at what’s on CTOs’ minds when it comes to generative AI as they look ahead. You can explore the full survey results, and filter by region, sector, and stage, via Sōzō Pulse.

For CTOs, coding has emerged as the killer app of the generative AI era — at least for now. It’s the use case that has seen the most adoption, generated the most enthusiasm, and delivered the highest ROI. In all, 90% of CTOs said they’re using generative AI in software development, 51% said it’s the use case they’re most excited about, and 34% said it has delivered the highest ROI. 

The results make sense to Prince Kohli, the CTO of Automation Anywhere, a software platform that intelligently automates and orchestrates processes using AI agents and robotic process automation (RPA) to eliminate repetitive tasks and improve productivity. While early experiments to use generative AI for software development with tools like GitLab brought underwhelming results, others have been more promising, specifically with new code: “It was very easy to generate templates for new functions,” Kohli says, “and within those new templates it was easy to get high-quality code” and save developer time. His advice to other CTOs: “You must experiment, and introduce your developers to some of the new tools. And you must start using it where it seems to be a good fit.” 

Trials and testing are a big part of Chief Security & IT Officer Allan Dembry’s approach to generative AI at Iyuno, which provides subtitling, translation, and dubbing services to the entertainment industry. Iyuno developers are using standalone assistants such as GitLab Duo and ChatGPT for software development, as well as tools integrated within platforms like Monday.com and Amazon Bedrock. “We’ve had mixed results,” Dembry says. After one trial that ran for several months, Dembry analyzed the impact on productivity: “While the results generally indicated a positive uplift, it was marginal,” he says. Instead of moving ahead with a wider rollout of those services across the enterprise, “we’re focusing our generative AI efforts on our own internally developed tools and operational processes.” 

Wil Bolivar, SoftBank Investment Advisers’ CTO, says the impact generative AI has had on software development is hard to overstate. But in light of the continuous improvements in the technology and the high rate of adoption, he expects generative AI will be embraced in a growing number of functions and use cases inside companies. “If you look back just six months and compare that to where generative AI is now,” says Bolivar, “most CTOs will tell you: ‘It’s much farther ahead now, and we’re seeing more value.’” 

Indeed, while software development was a clear No. 1, generative AI has gained broad adoption in a number of departments: 69% of CTOs said it is being used for product development; 61% for customer support; and 41% for creative/marketing. In terms of use cases, knowledge retrieval has generated a lot of enthusiasm: 53% of CTOs said they’ve embraced that use case; 41% ranked it as one they’re most excited about; and 32% said it’s delivering the most ROI.

Merve Hickok, founder of AIethicist.org, says adoption of generative AI across disciplines and departments is likely to ebb and flow as companies experiment and discover where the technology will be most useful: “You need to find the right use case for your organization, which can add value and sustain that added value in the long run.” 

Dembry agrees, saying: “There’s certainly not one ring to rule them all.” Some models, including LLMs, he says, lend themselves very well to text and task decision-based use cases. Others need to be trained and developed for specialist functions, such as video analysis and document interpretation and manipulation. “Standard models simply won’t give you the best results,” Dembry says. “Choosing the wrong model, or failing to fine tune, or even deploying in the wrong way, can result in poor results and even significant costs.”

Two years into the generative AI era, companies are not only adopting the technology more widely, but also growing more comfortable with it. This year 49% of CTOs said their teams are well-versed in the technology and ready to integrate it into their products, compared with 36% last year. Meanwhile, 44% of CTOs said their teams are enthusiastic but in need of training, down from 47% last year; 7% are interested but unsure where to start, down from 12% last year. And while a year ago, 4% of CTOs told us their teams were not interested or equipped to harness generative AI, no CTOs said so this year. 

“They’re building more confidence,” says Bolivar. “They’re starting to see that the product has gotten better. Initially there was doubt and concern about what generative AI could and could not do. Many who were somewhat reserved are now finding that it is helpful, and actually providing productivity returns.” 

Despite the growing level of comfort with and enthusiasm about generative AI, challenges that have plagued the technology since its inception remain top of mind. For example, 76% of CTOs said accuracy is their biggest concern, with cost following closely at 61%, and data privacy at 54%. When it comes to the specific challenges of integrating large language models into their products, CTOs cited unclear ROI (47%); insufficient technical talent and time (44%); poor quality of the end product (36%); unorganized data (34%); and security concerns and budget constraints/high cost both at 25%. 

When first working with AI-generated code, Kohli noticed “very subtle bugs.” What’s more, “we found that the subtlety of AI-generated bugs is worse than human-generated code.” Kohli attributes that to AI lacking “true intelligence.” Humans seem to find it easier to spot bugs in human-written code, as it seems to follow their own methodical approach, he says, but AI-generated bugs, on the other hand, can be more difficult to detect. The bottom line is that developers must review AI-generated code extremely carefully. “We’re very, very careful about any code that goes into production,” he says. “Don’t ever assume because it’s machine generated and looks reasonable, it’s correct.” But he remains very optimistic about generative AI as another helpful tool in the developer kit. “And as it gets better, its usage will naturally increase without being forced,” he says. 

As Dembry looks ahead at Iyuno, he envisions models becoming more accurate and hallucinations decreasing so that workflows can rely less on human intervention, freeing up resources for other tasks. “That will allow teams to focus on bringing real value to the business,” he says. 

Kohli echoes that sentiment: “It’s a very fast-moving field. More startups will come in and continue to reinvent the way code is being written. It’s just a matter of time.”