Trust

I've been thinking about trust a lot lately. Mostly because, as a software developer, I'm smack dab in the middle of an existential industry-wide crisis thanks to AI, and there's a lot of distrust of AI... much of it warranted, some of it not. Trust is critical to value. It underpins our monetary system (even if we were on the gold standard, btw). It's the primary driver of our ability to charge higher hourly rates (software devs) and to demand higher salaries (employees). It's at the foundation of the quality and longevity of our relationships.

It's very important.

But AI is not trusted... not very much, at least.

There are some areas that trust it a whole awful lot. Some in our industry are running 20-100 AI agents in parallel to build things. Uncle Bob (Robert Martin, author of Clean Code and a pillar in our industry) both trusts AI enough to not look at the code it produces at all, while simultaneously distrusts AI enough to impose multiple layers of strict constraints on the work that it does.

AI Trust

AI, in its current Wild West state, is reducing trust in everything it touches, on average. I say this as a propopent of the technology.

• We trust social media less (especially LinkedIn)
• We trust software less
• We trust video clips less
• We trust our employers less (are they going to replace me?)
• We trust our employees less (are they just using AI blindly?)
• etc

Trust in software

Software developers have been working on improving trust in the code that we write for decades. It's close to a solved problem. That doesn't mean we eliminate all bugs, because bugs are ultimately a reflection of the imperfections of humans and our inability to clearly articulate everything for all cases. But we have tools and techniques to get very close to zero bugs.

• Unit testing
• end-to-end testing
• Code reviews
• QA
• etc

All of these are still useful with AI, but less so.

No human is fully trustworthy, but they're more trustworthy than an AI agent. Each person has exponentially more skin in the game than a faceless LLM model. You can fire the human if they keep lying. Sure, you can use a different LLM model, but it doesn't care, and its corporate owners don't care that much... you're just a drop in the bucket. It's not life ending.

And humans are better at seeing the big picture, certainly making judgements on that picture for things that actually matter to humans.

So when a human writes a unit test, that unit test is much more trustworthy than a unit test that an AI agent writes itself.

Should we require unit tests from our AI agents when they write software for us? Absolutely. Does that mean we automatically get the same level of trust in the resulting software? Absolutely not.

Uncle Bob preaches "mutation testing" for this reason, along with layering on other types of constraints.

Quantifying trust

One of the best courses I took in college, which was now a really long time ago, was a course that they called Numerical Methods. Essentially, solving differential equations (DEs) using computers. The coolest emphasis was the concept that, with the right breakdown of the DE into its approximation (often an infinite sum series), we could guarantee a level of precision or error tolerance, bounding the error potential underneath a tolerance small enough for our use case.

Why not just calculate the result directly?

Well, you can't. That's not how computers work. Or in some cases, you could, but it would take a really really long time to calculate. You have to break it down into its approximation form, which means you will have some amount of imprecision (error), which means you have to confidently bound that error to get the precision you're comfortable with.

If we want to ensure a certain level of trust in the software that we produce, how do we quantity it? More specifically, how do we quantify it in the realm of AI?

We've already been quantifying trust at the system level, typically in the form of uptime SLAs (99.9% is "three nines", for example).

But that's still a backward looking measure, ultimately. How can we quantify trust so that we can have some assurance before deployment?

Test coverage comes to mind, but it's not super useful if we can't trust the tests to begin with. Mutation testing helps. More layers of tests help. Performance tests? Load tests? Fault tolerance tests? "Creative user" tests?

Is testing all that matters? It matters a lot, sure. "If it's not tested, it doesn't work" is an old saying in the industry. But there are other ways that customers lose trust in the software they use and in the software dev shops they employ.

One way is the increasing marginal cost of additional features. It's cheap to add new stuff early on. Later, as the application and codebase matures, it gets more and more costly (takes more time and effort) to even add a simple new change. Testing helps to reduce this cost a lot. But "code quality" matters, as well.

So we also need to quantitatively measure code quality, which is a bit subjective, but can we find any close-enough approximations? Cyclomatic Complexity certainly comes to mind. Again, Uncle Bob mentions something called CRAP (which stands for something, but he doesn't know exactly what).

• Coupling
• Function Complexity
• Side effects
• Naming consistency
• Tight domain kernel
• etc

But none of these things actually quantify TRUST. They're instead just tangential approximations to trust in their small spheres.

I think the quantification of trust ultimately is the same everywhere: money.

Why does gold have value? Because we trust that it's worth that much now and in the future when we want to exchange its value for something else that we need/want.

Why does fiat currency have value? Because of the above, and for the banks making the loans (creating the money) its because they trust that people taking out loans will generally pay them back.

How do we know that the software services we provide are trustworthy? Because people keep paying us.

There are other approximations to trust, of course, like surveys of feelings, but nothing is as accurate as cashflows.

Ok, but that still doesn't tell me if the work that I'm doing is trustworthy in the long-term. Maybe I'm just getting lucky. Maybe I'm a swindler. In what I can control while producing my work, what should my quantification of trust look like?

I'm asking questions in this post. I don't have answers yet.

But we've seen other examples that could work for this, in the way that LLM models themselves quantify their own trust: benchmarks.

Good-enough software

But once we have some decent quantification of trust, something that we can control by the work that we produce, then we can finally get to the point where we can guarantee a certain level of quality.

Suppose we have a Software Trust Score (STS) where lower is better: 5 is better than 10, 0.1 is better than 1.0, etc. And suppose that we can measure this quantitiatively, with computers. It's ultimately a simple algorithm. Well, then, we can automate that entirely, and require any human or AI agent to keep working on the work they produce until their work satisfies the arbitrary STS that we've imposed.

For projects where quality (trust) is more important, we can turn the dial to make sure the STS gets met, likely costing us more in tokens, time, and other input costs. But we can automate it.

And if we can automate it, then we can also turn up the dial in the number of parallel agents that we have running for us, writing our code. Once their production satisfies the STS, the software is "good enough" for our given constraints, and we can stop spending money making it better than it needs to be and move on instead to the next set of features valuable to the business.

Craftsman to Engineer

This will need expounded upon in the future. Before AI, those of us who cared about quality were akin to craftsmen. We were writing "hand stitched leather" code.

And there's still a place for that post-AI, just as there's still a place for old-school craftsmanship in other domains that have since been replaced by industrialization for the masses (mens suits come to mind).

But as craftsmen, we still worked for the businesses that hired us to produce work that was valuable to the business. Well-crafted code was important to the business only until its marginal increase in quality cost more than its marginal increase in business value for that extra quality.

Now with AI, we can build full-fledged software factories. The craftsmen must become the inventors and the engineers, crafting the machines and processes of the factory rather than the outputs of that factory.

• Measure and enforce quality standards
• Ensure safety
• Optimize costs and improve efficiency
• etc

All the while, a factory engineer still has intimate knowledge of the end product, but they aren't pulling out screwdrivers and hammers to build each product by hand after a couple early prototypes.

What does this have to do with trust?

You can't setup a factory with automations that aren't trustworthy. Your quality standards must be measurable and enforceable. Your safety protocols must be well-defined and adhered to. You can't spend unlimited amounts of money on energy and scrap and materials or you won't end up making profits.

So finally, we can measure the outputs of our different factory versions against a suite of standard benchmarks to see if our factories themselves are improving or not, on average. Continuous Improvement of virtual factories.

• How accurately did we solve the software problems in the benchmarks?
• How much did it cost us?
• Did we break anything?
• How many bugs did we have along the way that didn't get caught?
• What's the final code quality?
• How did velocity change as the project grew more complex?
• etc.

We're in the early days of software factories. Much of it remains theoretical, but it's moving fast. Despite the challenges, everything appears to have quantitative potential, which means it's ultimately objectively solveable, which means it's very likely inevitable, only bound by cost-to-value comparisons.