The Real Cost of AI Depends on the Job [Raw Session]

00:00 — Opening / Tokens Are the Visible Cost

Hey welcome back to Slow Builds. So thinking about AI cost but not in the usual way people look at it. Most of the time people are talking about tokens.

It’s always tokens and this comes up a lot for me too because at work and paying for personal stuff and things like that. So how much does a prompt cost? How much does the output cost? Which model is cheaper? How many messages do you get.

And that makes sense because that is part of the part right now that we can actually see in front of us and what people are going through between using ChatGPT, Claude, and just using some APIs.

And the cost shows up. It shows up as tokens, credits, usage limits, depending on what model you’re using. But I keep wondering if that’s just way too simple right now. Because not all AI usage is same kind of usage.

A person asking a chatbot questions is one thing. A company training a model is something else. A doctor using AI to double check symptoms is different from a video generator running overnight. A developer fixing production outage is different from a developer asking agents to slowly work through new features or backlog bugs.

Those are all AI but they are not the same kind of work. Some maybe the future cost of AI is not one price. Right now it’s not one price either but I imagine it’s going to be broken up a lot more granular.

Maybe it depends on the job itself and maybe that’s where this whole thing starts to split apart a little bit.

01:45 — What Sits Behind the Token Price

So right now tokens are the easiest thing to talk about because they are measurable. Input tokens, output tokens, your context window, your API bills, your message limits. That is the visible units that we’re all seeing on our subscriptions, on our work invoices.

But behind every token there’s a bunch of other costs that we do not really think about when we are just typing in the box. Which model is being used? What hardware? How much power consumption? What is the speed that we expect a response? How reliable? How consistent? What privacy is involved?

There’s an amount of reasoning that we need. How many times can we try the same thing? They determine the right answer. There’s an amount of checking going on in the background.

So two AI tasks might both use tokens but they’re not really the same cost. One job might need the best model available right now with very little room for error. Another job might be fine with a small model slowly working away in the background.

And that distinction really matters because it really determines the cost that’s involved and what outcome and what speed and timing and consistency is required. Because if we’re only talking about a token price we missed the bigger question, what kind of work is the AI actually doing?

03:20 — Some Jobs Need the Best Model

And I think this is kind of where AI usage starts to split.

And then AI is messing this one up.

Some jobs need the best model.

Some jobs do not.

If you’re making a serious architecture decision in software or thinking through a medical case or reviewing something legal or financial, you probably want a stronger model, a very consistent model with a strong background, vast knowledge, and very precise.

You might want more reasoning, more checking, better context, and maybe a human involved along the way for checking.

But a lot of everyday AI work does not need that.

Summarizing a short email, renaming files, sorting notes, pulling dates out of documents, clean rough text, organizing simple lists, a list does not need the most expensive models in the world.

It could actually run on a local model running on your machine for free. Well free like you do have to pay for your your energy and consumption that way but other than that you’re not paying tokens and that is where I think the future gets way more interesting because of using one big model for everything more of this is probably becomes routed.

Cheap model first, better model if needed. Specialized model for narrow work. Local model for private or repeatable work. Expensive only when the task actually deserves it.

That feels different from how people often use AI right now, where you just pick the best model and use it for everything. But that might not make sense long term.

It is like using a full-size truck to pick up groceries. Just one bag. It works but it’s probably not the most efficient way to do it every single time.

05:03 — Urgency Changes the Cost

Urgency, like the biggest piece for me is the urgency because speed costs money, reliability costs extra money, low latency costs money. If AI needs to answer right now, that’s a different kind of system.

A surgeon using AI cannot wait around for background job to finish. A self-driving car cannot send something away and hope the answer comes back before the light turns red. A robot in a factory, a security system, or a production outage all have that same kind of problem.

They need low latency, they need consistent results, they need them fast and accurate. The answer has to be close. It’s probably going to be more expensive, but not everything is like that.

A family doctor using AI to think through diagnostics. Diagnosis may need accuracy, but not necessarily a half second response. A developer building features or working on bugs or non-production bugs may not need every answer instantly.

Business owner asking for a report might be fine getting it in the morning.

An AI agent reviewing documents overnight does not need to act like a live chatbot.

It doesn’t need to have the answer instantly.

It can run in the background slowly.

You can run on a local model like video generation, simulations, big code cleanups, document reviews. Like a lot of these things, they don’t need to be overly expensive and overly consuming.

So the question becomes pretty simple. Does this AI need to be fast or does it just need to be done?

That one question changes the cost. And I think a lot of future AI work is going to fall into the category. Not urgent, still valuable, but you don’t need it right away.

You don’t need the best model. You may not even need, like I said, you could probably run it locally. You don’t need to be outsourcing it through an API and burning through tokens. A lot of it can just be running in the background on your own machines and your own system, maybe even on your own laptop.

I have a Ubuntu running out in the room and it does plenty of stuff for me that I don’t need to spend money on besides the energy that it takes to run it.

07:32 — The Can Wait Category

I think the can wait category is much bigger than people realize because there’s a lot of valuable work that does not need to happen in real time.

Reading documents, checking contracts, looking through old notes, watching for changes, generating reports, testing code, cleaning data, comparing options, building prototypes, reviewing logs, summarizing meetings, most of this does not need instant response.

I wanted to read through those because there’s a lot of different cases where when you really think about it you could throw it into a job. You don’t, as long as you get an update and overview and a response at some point and you know you’re going to get it, it’s almost like set it and forget it type thing.

You don’t need to sit around waiting for the prompt to return the results instantly

And that is probably going to be 80-90% of all the AI work that happens.

It just needs steady background compute.

And if it can run in the background, it can probably be cheaper.

It can use different models, it can run at different times, it can be queued, it can be batched.

Like, just, I don’t know, highlight this.

Because like, even think about like the power grid where different times cost different money for energy. So if you have a bunch of things, you can hold off and run it at the low times so your energy is cheaper.

Maybe you live in a place where you can collect a lot of solar energy, enough that you can run a machine off of.

So during the day, you’re collecting the energy and at night you run the models.

There’s all kinds of ways around this.

You can run when compute is cheaper or less busy.

Look, I had that right in the next thing.

That is a very different cost model than sitting in front of a chatbot waiting for an answer or to appear word by word.

09:21 — Cheap Background AI Changes Usage

And I think that changes how we use AI because when something’s expensive and immediate, you’re careful with it.

But when something is cheap and can run quietly in the background, you start using it differently.

You start asking it to watch things, check things, compare things, tell you what changed, look through files, review code, prepare drafts, clean up messy work.

Not because it’s dramatic, but because it becomes very useful.

So you can optimize your life, you can improve your systems, you can improve your workflows, you can produce more output without breaking the bank or without like, you know, burning down a forest.

Like you allow these things to run slowly in the background on their own and it’s slowly improving your life without you doing heavy lifts, without you spending a lot of extra.

but a lot of little things are adding to the final product.

10:21 — Placement Matters: Local, Cloud, and Remote Compute

And this is where the infrastructure part comes in, but I do not think the infrastructure is the whole point.

The point is placement.

Where should the AI job run based on what job is needed?

Some AI will run locally on your phone, laptop, home server, inside a company network.

I’m gonna highlight this again because it’s funny

Now I’m seeing more online in the last few days or a week about running, getting out of the cloud, bringing things back in locally into companies and infrastructure within their organizations.

Not massive, huge, like it used to be, like massive in-house data centers and server rooms, but more run a lot of things locally rather than outsource everything ’cause it does cut down on the bills.

maybe they have their own power sources that they can control and they can maintain and they’re not relying on others.

Maybe they have enough RAM and GPU running around in their company that they have down times.

So there’s lots of different things and the fact that we went from massive computers down to small computers and laptops and then we moved to the cloud and like it always shifts.

and I do other videos about that where the value is moving, what the world values now and how it constantly moves around.

It all depends on the bottleneck and where people’s attention lay.

So another video is coming on that.

But it’s just always interesting how everything’s always changing and a lot of things always come back to the same thing in the end.

So like I said, and companies are doing that.

They’re moving it back into their local networks now.

And that makes sense for private work, simple work, repetitive work, or even just personal stuff.

Stuff where you do not need the biggest model and maybe you don’t want the data leaving your device or you want to keep it contained within your own privacy.

Some AI will run in the cloud and it will always run in the cloud.

And that makes sense for harder tasks, larger models, business systems, shared tools, anything that needs more compute than your own device can handle.

And in some AI might run more remote places where power is cheaper, cooling is easier, where infrastructure is built specifically for background compute.

Maybe that is near a hydro, maybe that is near a nuclear, solar heavy regions, maybe one day some of it is space based compute.

Whoever thought that was going to happen.

But the point is not space.

The point is that not every AI job needs to run in the same place.

If it needs speed, keep it close. If it needs privacy, maybe keep it local.

If it needs massive models, send it to the bigger infrastructures. If it can wait, send it somewhere where it’s cheap and can run in the background at low cost.

And that’s your cost splitting, you’re breaking it up based off the role that’s required and the output that’s needed.

13:31 — AI Cost Becomes Routing

So maybe the future is not one simple AI price, maybe the future is routing.

The system asks what is the task? How hard is it? How urgent is it? How private is it? How accurate does it need to be? Can it use a small model? Does it need a frontier model? Can it wait? Can it run locally? Can it run overnight? Can it run somewhere cheaper?

And then the work gets sent to the right place and maybe there’s multiple places along that workflow flow before you get the final answer.

And that’s a different way to think about costs because today we mostly see the front end. We type in the box we get the answer.

But behind the scenes the future might be much more layered. Small models doing small jobs, big models doing hard important jobs, local doing private, fast systems doing very urgent, and background just doing the slow jobs, the invisible work and the cost depends on where the work lands.

That feels like a more realistic version of where this goes.

Not one AI, not one price, not one model doing everything.

More like different layers of intelligence used for different kinds of work.

And that’s what I said before. This is a me off script again. I don’t think there’s one winner in AI. Yeah, the AI model, the data model. I talked about that, but they all seem to do a little bit different ones gonna be better at testing one’s gonna be better at documentation and content one’s better at writing code and so it all comes into play where I think they’re all going to be part of the process rather some are like I said some are gonna run local some are gonna be small some are gonna be big for the privacy some are gonna have to go out to where we had the data access somewhat like it’s going to be distributed across many places depending on the flow and requirements of the request and this so and this is why I guess get back to this now why cheap background AI matters

15:46 — Why Cheap Background AI Matters

this matters because because once some AI gets cheap enough people stop thinking about it the same way.

They stop asking should I spend tokens on this and they start asking can this just run in the background.

That’s a massive shift because a lot of useful AI work is boring.

It’s not always a big impressive demo.

It’s more like check this, watch that, summarize, clean, find anomalies, look for patterns, tell me what changed, look through files, review code, run tests, prepare drafts, and you do it all while you sleep.

There’s no, like you might set a deadline, but you’re not overly concerned.

That kind of AI could become normal if the cost gets low enough.

Not because every model is cheap and not because AI is free, but because the right kind of work gets routed to the right kind of compute.

And that’s probably where the usage grows.

Not only from people chatting more, but from AI running quietly around the edges of everything.

Watching, checking, sorting, comparing, cleaning.

Doing the slow work that people often avoid because it’s boring.

And boring always wins.

Boring, repetitive, it just requires too much attention.

So I think about reading, studying for reading a biology book for your exam and falling asleep.

It’s that kind of work.

That is the part that feels and I’m sorry about the biology part. I just had to do that for university myself when I was younger and I remember that specifically and that is the part that feels easy to underestimate.

17:26 — The Real Cost Depends on the Work

So I do not think the real cost of AI is just token counting.

Token count matters but it is only one piece of it.

The real cost depends on the kind of work being done.

Some AI is expensive because it needs speed, because it needs the best model.

Reliability needs a huge amount of data to get the right context.

And it also depends on how high the stakes are.

Like I said, doctors, driving, flying, any kind of high risk, instant answer type thing is gonna be expensive.

But some AI probably becomes cheap because it can wait.

Smaller models, local, running in the queue, overnight.

It can run when triggered rather than always on.

Run when power’s cheaper rather than on all the time.

And that part I think is easy to miss.

We talk about AI costs like it’s one thing, but AI usage is not one thing.

Like I said, like surgeons, family doctors, a developer doing production versus developer building a feature.

Students summarizing notes, the company’s analyzing documents, lawyers preparing briefs, accountants doing tax returns.

A lot of it all depends on the current situation and what’s expected of the result.

So maybe the future is not one AI price, maybe it’s different layers of intelligence with different costs, fast, cheap, local background, high trust, good enough.

And the more those layers split apart, the more AI usage changes.

Because once the cheaper layers get good enough, people stop treating AI like something they only use carefully.

They start treating it like something that is always working around them.

19:20 — Watching AI Become Embedded Everywhere

As part, and that is the part I’m trying to watch, and I want to see, I’m actually starting to see it a little bit more.

Not just what the best model can be, but what happens when the right model is in the right place, at the right speed because becomes cheap enough for the job.

And I really see this because my initial reactions, I still think are valid where this is gonna change everything, the world’s gonna look different and AI is gonna change and it already has.

But I really believe that it’s the people who are on top of it and learning it and in it and seeing how it can be used I think is going to have a big difference in what I always looked at taking jobs it’s not just going to take jobs it’s going to create new jobs obviously there’s going to be less of what we see today so what people are going to school for what they’ve been doing for the last 10 20 30 years is going to change but I do believe everything’s going to change for a good way there’s going to be some downfall but there’s going to be a brighter future in my mind and I can see how AI is going to be embedded in everything which it already started has and is getting more and more constantly.

This is not a normal technology that’s just coming out.

It’s world-changing and we’re seeing those effects already.

So I believe it’s going to be broken up. People are starting to use it differently.

We’re going to see it spread out sort of like how we saw a cloud compute, SaaS companies. That was a foreign form before Salesforce, but now it’s the norm and we’re rolling into that I believe.

So anyway I’m very interested to see what happens and thanks for watching bye.