Harness pitches AI agents as your new DevOps taskmasters • The Register

At its Unscripted event in London, DevOps company Harness presented its latest AI-driven modules, including an AI pipeline builder, AI test automation, autonomous code fixing when builds fail, AI AppSec (application security) and even AI-driven chaos testing, where resiliency is tested by introducing random failures.

According to Harness, software teams spend only 30-40 percent of their time on planning and coding, the rest being consumed by testing, securing, deploying, and optimizing applications. The implication is that big productivity gains are possible by extending AI assistance to these post-coding processes.

Harness CEO and co-founder Jyoti Bansal presents at Unscripted in London

Given that AI is non-deterministic and vulnerable to issues including hallucination and prompt injection, is it safe to entrust security-critical DevOps processes to the technology?

CEO and co-founder Jyoti Bansal told us that hallucination is more likely with generic use of large language models (LLMs) than with the Harness AI agents. Given a task such as creating a build and deployment pipeline, “our agents will break the tasks into smaller tasks, and we have purpose-built agents for those smaller tasks,” he said. “These agents also cross-verify each other’s output.”

Another key factor is context, he said, knowledge of the organization, and of “the things you’ve done in the past, your builds, your security tests, your code changes, your services and their dependencies. That is used by the agents… so you don’t have inaccuracies and hallucinations.”

Further, he assured us, “none of this is done without human input… our AI is not doing the deployment to production. Our AI is creating the deterministic pipeline to do the deployment to production.” Non-determinism cannot be completely removed, he said, but after the AI has created an automation, “you audit it, you review it, its compliance, its governance… We are not doing the deployment through AI, we are creating the deployment pipeline and automation through AI. Then it is deterministic and repeatable, because there is no AI when you’re running it.”

The matter of human checking, sometimes described as the “human in the loop,” is difficult, though. During the keynote for the event, Bansal described how non-experts who could not previously create user interface tests could now do so by describing in plain English what was required. He also said that the use of AI-generated code meant that the volume of code might be as much as four times greater than before, making it hard for a human to check every line. “That’s why you need the check and balance process to be much more robust,” Bansal told us. “The entire process has to improve significantly, all the testing, deployment, rollback, governance, compliance, everything around it.”

Given that, on the Harness platform, all these processes could themselves include AI, does that mean we are asking AI to check its own output?

“You can have AI check what AI has done, but I would always recommend that you need two different AIs. Don’t trust one AI to do both, the way you wouldn’t trust one person to do your accounting and the same person to do auditing.”

A related issue is that the human skills needed to verify code or complex processes may themselves be harder to find if people become more reliant on AI.

“I think engineers need AI taskmaster kind of skills,” Bansal told us. “Good engineers have to understand what to ask AI, and how to do it better and properly… those are becoming the top skills.”

Using AI, he said, is “an iterative process. You ask AI something, you review it, then you ask it to do something, then you review it. That becomes your skill, and I think that will be the core of how software engineering is done.”

It is reminiscent of the conclusions in the latest Google DORA (DevOps Research and Assessment) report, and indeed the DORA research was referenced in the Unscripted keynote.

There is one piece of good news for AI skeptics. AI in the Harness platform is optional. “You can turn off everything in AI, and you can also turn it on in pieces,” Bansal said. “You can say, I’m fine for AI for testing but I don’t want AI for my security runtime protection. You can also do it for different teams and applications… there are different degrees of comfort and skepticism, and we allow for that.”

The Harness DevOps platform runs on Kubernetes, with a control plane either in the cloud or on-premises (80 percent cloud, according to Bansal), and Delegate workers, which always run on-premises or on an organization’s virtual private cloud. There are free plans for small teams, an Essentials plan for up to 500 users at $30 per user per month, and an Enterprise plan typically for $100K-200K annually. ®