**I Gave an AI Control of My Smart Home — Should You? — Transcript & Summary | SozAI**
Source: https://sozai.app/transcript/ai-control-smart-home-should-you/

Exploring AI control of smart homes via MCP servers, risks, setup, and an AI agent that self-heals Home Assistant automations.

## Key Takeaways

- Giving AI control over smart home automations can enable self-healing and proactive maintenance.
- MCP servers are the critical technology that bridges AI models with real-world device control.
- Security risks must be carefully managed by limiting AI access and using secure network exposure methods.
- Manual approval of AI actions is a crucial safeguard to prevent unintended or harmful changes.
- Users should weigh the benefits of AI automation against the potential risks before granting control.

## What the video covers

- The video demonstrates how an AI agent named Alfred can detect and fix broken automations in a Home Assistant smart home setup.
- It explains the concept of MCP (Model Context Protocol) servers that enable AI to interact with and control smart home devices beyond just conversation.
- Two types of MCP servers are discussed: the official limited Home Assistant MCP server and the more powerful community HAMCP server.
- The trade-offs between capability and security risks are highlighted, emphasizing careful exposure of devices and tools.
- Key risks include AI taking unintended actions, prompt injection attacks, expanded attack surface, and exposing the home network to the internet.
- The video covers safe setup practices using Nabukasa to securely expose Home Assistant to AI without compromising network security.
- Demonstrations include connecting AI models like Claude and ChatGPT, and how Alfred reads logs, diagnoses issues, and proposes fixes.
- The importance of retaining manual approval for AI changes is stressed to maintain user control and safety.
- The video provides a step-by-step tutorial on installing and configuring MCP servers and AI agents within Home Assistant.
- Viewers are encouraged to decide their own comfort level with granting AI control over their smart home systems.

## Chapters

1. 00:00 Introduction: AI Detects and Fixes Smart Home Issues
2. 02:08 Understanding MCP Servers and Risks
3. 04:10 Types of MCP Servers: Official vs Community
4. 06:30 Security Risks and Best Practices
5. 08:49 Installing the HAMCP Custom Component
6. 11:00 Connecting AI Models and Tools
7. 13:00 Configuring Entity Visibility and Permissions
8. 14:30 Demonstration of Alfred AI Agent in Action
9. 19:46 Conclusion: Balancing Control and Safety

Answers

## Questions about this video

What is an MCP server and why is it important for AI smart home control?

An MCP (Model Context Protocol) server is an open standard that allows AI models to interact with and control external systems like smart homes. It enables AI to read data and take actions, bridging the gap between conversation and real-world control.

What are the main security risks of giving AI control over a smart home?

The main risks include unintended AI actions that could damage the home, prompt injection attacks where malicious data tricks the AI, increased attack surface from exposing many devices, and potential exposure of the home network if not securely configured.

How does the AI agent Alfred ensure safe changes to the smart home?

Alfred detects issues and proposes fixes but waits for the user's manual approval before applying any changes. This approval step is a key safety measure to maintain user control and prevent unintended modifications.

## Full Transcript — Download SRT & Markdown

00:00

Speaker A

So, for about a week, my front yard camera alerts quietly just stopped working. Someone would pull into the driveway, and my phone said nothing. And here's the part that gets me. I didn't catch it. My house actually did. The AI

00:14

Speaker A

agent I created, I've named him Alfred, reads my home assistant logs on a schedule. He found the automation behind those alerts was throwing the same error over and over every single time somebody drove up to the front of the

00:26

Speaker A

house. It was actually trying to send an alert to a phone that, as far as Home Assistant was concerned, didn't exist anymore. So, every alert just failed silently. But Alfred then did the part that I actually care about. It didn't

00:40

Speaker A

just report the problem to me. It showed me the exact one-line [music] fix and asked if I wanted to apply it. And as soon as I gave him permission, he made the change and rearmed the alerts. Later on that day, when Amazon came to drop

00:53

Speaker A

off a package, my phone actually lit up again. I didn't have to open a config file. I didn't touch one line of YAML.

01:01

Speaker A

My smart home found its own broken automation and repaired it. I just had to approve the fix. But that only works because I gave it something first. So to let an AI fix my house, I had to hand it

01:13

Speaker A

real control. The ability to read what's wrong and, if I'm not careful, change things when I'm not looking. Would you give an AI the keys to your smart home [music] if it meant it could actually fix itself?

01:26

Speaker A

[music] Hi there neighbors. I'm Ryan the tech guy, and welcome back to this smart house. So before we go any further with today's tutorial, I want to be precise because I want to make sure you actually understand what we're doing here. I did

01:37

Speaker A

say that my Home Assistant self-healed, but what you actually saw was an AI detecting a problem, diagnosing it, and proposing a fix to me. Then it waited for me to actually approve that change.

01:49

Speaker A

That approval step isn't something that I forgot to remove. It's the line that I choose to keep. And by the end of today's video, you'll understand why.

01:56

Speaker A

And hopefully, you can make a decision where to draw your line. Because none of this works unless you are willing to connect your Home Assistant to an AI agent and give it real access. That connection runs through something called

02:08

Speaker A

an MCP server. And getting it wrong is how you risk handing a chatbot the keys to your front door lock or your garage doors. So, here's the plan. First, we're going to take a look at what an MCP

02:19

Speaker A

server actually is and the honest risks that come with setting one up. Then, I'm going to show you how to set this up in your Home Assistant environment, exposing it safely through something like Nabukasa, and connecting it to the

02:30

Speaker A

AI of your choice. I'll show you Claude and ChatGPT along with my own AI agent that I use. Then, what it can really do once it's wired in, including that log-watching loop that you just saw. So, by

02:42

Speaker A

the end, hopefully you'll be able to build this yourself and decide how many keys that you're comfortable handing over. And don't forget, if you want to follow along, you can access the full blog post at the link down in the

02:52

Speaker A

description or the QR code down here below. So, to understand how this all works, you're going to need one concept, and that's a new concept that's come along with AI and really impacts everything that it touches, not just

03:03

Speaker A

smart homes. So, normally an AI like Claude or ChatGPT can only do one thing, talk. You type, it types back. It can describe how to fix a Home Assistant automation, but it can't actually reach into your home and make the changes

03:16

Speaker A

itself. It's kind of like looking through glass. And that's where the MCP server comes in, or model context protocol. This is what actually breaks that glass. It's an open standard that was released by Anthropic back in late 2024. It is now adopted basically

03:31

Speaker A

everywhere. That's what lets an AI connect to an outside system and both read from it and take actions on it.

03:38

Speaker A

Some people call it the USB-C for AI, one standard plug between the model and all your stuff. It basically works by exposing a set of tools and then describing to the AI how it can use it.

03:49

Speaker A

An MCP server is the adapter that sits on the thing that you want the AI to reach. You'll find them on all sorts of systems like you can get one for NN, you can get one for Adobe After Effects,

03:59

Speaker A

pretty much anything you can think of, you can run an MCP server and give AI control over. So, if you put one of these in Home Assistant, suddenly the AI isn't just talking about your house, it's actually in it. So, one honest

04:10

Speaker A

warning before you go searching because this tripped me up as well. So, to add to the confusion, there are actually two different types of MCP servers that are available when you search up Home Assistant MCP.

04:20

Speaker A

So, there's the official one that's built into Home Assistant, and it's called model context protocol server.

04:25

Speaker A

Pretty descriptive. It's solid, safe, and is limited on purpose. It can only see and control the entities that you deliberately expose to it. It can flip on a light that you've allowed, but it can't read logs and it can't rewrite an

04:37

Speaker A

automation. If you're familiar with some of the Home Assistant voice features, that's what this actually uses in order for it to communicate with an outside LLM. So, the one that I'm using is a community project called HAMCP, and it goes much further. Reading logs,

04:53

Speaker A

editing automations, and managing add-ons. It has over 80 tools. That extra reach is the only reason why Alfred could actually catch that error in the automation and fix it. The official server couldn't have done that.

05:07

Speaker A

And that right there is the whole trade-off of this video. More capability means more keys that you have to hand over. The safe limited one can't automatically fix your house. The powerful one can, which means it can also break it. So, let's be clear about

05:21

Speaker A

the risks before we build anything. So, the first risk is that it acts. It doesn't just answer. The moment you connect a right capable MCP server, the AI can change things like toggling devices or editing configuration files.

05:35

Speaker A

A wrong move isn't just a wrong response. It could be a damaging action in your actual home. Risk two, and this is the one that most people miss, is prompt injection. Your AI reads text from your house, device names,

05:49

Speaker A

notification messages, and log entries. If any of that text contains instructions, say a device that somebody else named something sneaky, a naive agent can be tricked into following them. The AI can't always tell your data apart from the command that's actually

06:03

Speaker A

hidden inside the data. And that's not science fiction. It's the central security problem of every tool that uses AI right now, which is why the next one really matters. So at risk three, we're talking about attack surface. Every

06:17

Speaker A

single entity that you expose, every tool you grant, that's one more thing that can go wrong if the AI gets confused or manipulated. 80 tools is a lot of surface. So the one rule that I follow is expose the least amount of

06:30

Speaker A

tools or devices that you need. My door locks, my garage door, or my alarms, those are walled off. The AI that fixes a broken notification has no ability to unlock my front door. And that's on purpose. In risk four, you could be

06:46

Speaker A

potentially opening a door onto your home network. For a cloud AI to reach your Home Assistant, your instance actually needs to be reachable from the outside world. If you do that carelessly, then you've published your entire home to the internet. So, we're

06:59

Speaker A

going to approach this carefully, which is the whole reason that I route things through Nabukasa in the next section as opposed to poking a hole in my firewall.

07:07

Speaker A

Now, of course, if you don't have Nabukasa, you can do this another way using something like Cloudflare Tunnels potentially, but we won't cover that on today's video. The official server keeps the power small and safe, while the community server makes it large and

07:21

Speaker A

useful, but puts the burden of security on you. So, if you keep in mind two things, it'll keep you

07:35

Speaker A

point forward follows those two rules. So, let's head over to the computer and set it up safely. All right, so now we're going to install and set up the HAMCP server. So, the easiest way to do this, first of all, is using Hacks or

07:49

Speaker A

the Home Assistant community store. If you're not familiar with that, I've got a link up here to a video I've done in the last year explaining how to get it set up. It's super powerful. So, if you haven't looked at it before, I

08:00

Speaker A

definitely would take a look. All right. So, the easiest way is to either use the link that I have below or head out and just Google HA-mcp server and it'll take you to the GitHub repository. From here, we can scroll

08:14

Speaker A

down. This, of course, is the GitHub repo where it'll tell you all the information about it. So, I'd recommend reading through this if you haven't already. So, the easiest way is to click the My Home Assistant button here, or

08:24

Speaker A

you can follow through the specific instructions down below. I'll take you through that one cuz it's really not that much more difficult. We're going to copy this URL here. Then we're going to head back into Home Assistant, pop into

08:35

Speaker A

the hack store, go to the top right hand corner, and select custom repositories. We're going to paste that URL in, and then under type, change it to integration. Then click add. It should show up on your list here. So, we're go

08:49

Speaker A

to the top here, we're going to type in MCP. And then at the very bottom, we'll find the HA MCP custom component. Click on that. Click in the bottom rightand corner the download button. Download.

09:01

Speaker A

So, of course, once you download a new integration, you'll need to go ahead and reboot Home Assistant. And then when you go back into Home Assistant, go down to settings, devices and services, and then on the integrations tab, click on add

09:14

Speaker A

integration. So, under select brand, we're going to type in HA-MCP. And the custom component comes up. We'll click on that. And of course, you have two options here. We're going to do the first one, which is the full server. So,

09:27

Speaker A

we're click on that one. And if you note here, it's going to actually run the entire MCP server inside Home Assistant, and go ahead and set up all of the exposure through Nabukasa using the Home Assistant web hooks because before you

09:43

Speaker A

actually had to install an add-on, which is basically a Docker container, and then set it up through there along with an additional component to allow it to work through Nabocasa. Now it's all in one and very streamlined, so it's even

09:54

Speaker A

easier to set up. So from there, we're going to click submit and it's going to create an item right here, which you can add. I put mine in systems. Hit finish.

10:04

Speaker A

So once you have it actually installed, the next step isn't as clear, but we got to go back out to the list of integrations, find it in our list, which of course we can search at the top here.

10:15

Speaker A

Click on the custom component, and then click on the gear icon here to go into the configure. Now, if we scroll to the top, we're going to see a number of connect URLs. Of course, these are blurred out on screen cuz they would

10:26

Speaker A

connect you to my system, and I don't want that. The top one has my Nabukasa URL already in it. So, that's the one I'm going to use because it's externally accessible. Down below that, we have some options for authentication.

10:39

Speaker A

I'm going to use a secret web hook URL, which again, if you happen to know that URL, you would have the ability to interact with it, but of course, it's a bunch of random strings, so very low probability. somebody's just going to

10:49

Speaker A

guess that. You also have options to sign in with Home Assistant using OOTH. Um, or you can use the legacy OOTH, which we wouldn't really recommend. Now, there's a ton of other options down here, and we're not going to go through

11:00

Speaker A

them because for first- time users, it's just easier to start with that, but we're going to grab we're going to go ahead and rightclick on this and copy the link address so we can use that here later on. And then we'll click submit.

11:13

Speaker A

So, the options are saved successfully. So for our first security landmine, we were going to talk about the string that it creates. That of course is a random generated string, but it's not 100% secure. If somebody happened to get a

11:25

Speaker A

hold of that, they could then get in and modify your Home Assistant instance. So make sure to treat this just like you would any other password. Now, if you're not familiar with Nabukasa, it is Home Assistant's official way of routing into

11:38

Speaker A

your system. It's done all through their servers and systems, and it costs like five bucks a month. But one of the features that you get with it is what's called web hooks, which is the ability for you to send a command through their

11:50

Speaker A

server that then triggers an automation inside of Home Assistant. So this is backpacking on that same concept. So that way you don't have to open up any ports in your firewall. So now the server is actually up and running in the

12:00

Speaker A

Home Assistant side. Let's switch over to an AI of our choice to get the connection set up. So I'm going to show you the two most popular options, Chat GPT and Claude. Now Claude is my AI of choice. I spent about 100 bucks a month

12:14

Speaker A

on it for the max plan, but that's not necessary here. So, let's go ahead and start off with Claude. We're going to go ahead and open up our Claude desktop app. Go into settings and connectors.

12:25

Speaker A

So, at the top, I'm going to click add and add a custom connector. So, at the top, we're going to type in Home Assistant. And then we're going to take that URL that we generated in the last step and paste it here in the MCP server

12:37

Speaker A

URL field. Now, remember this is contains all the information you need to get to your system. So make sure you keep it secure. That's why it's blurred.

12:45

Speaker A

So after that, we're going to click add. And then it's going to ask us to go ahead and connect and authorize. So this is going to exchange all the keys necessary and get us authorized to be able to use this permanently.

12:58

Speaker A

So that's it. Since we've already pre-authorized everything using that web hook, we don't need to do anything else.

13:04

Speaker A

I'm going to click open claude. And now it's going to pop back up and show me that it has access with this tool. So now I can go down through here and actually select different tools and it breaks them into readonly tools and

13:16

Speaker A

write delete tools. I would recommend always to start off with making sure everything is marked as needs approval.

13:21

Speaker A

And this is where you can go in and scope different things that you don't want the AI to have access to. So if you want the AI to only be able to read your environment, then you can click on where

13:30

Speaker A

it says needs approval here under write delete tools and mark blocked. Then in no way, shape or form unless you go change that will the AI be able to use any of those tools. You could, if you're really feeling risky, and I don't

13:41

Speaker A

recommend this at all, go ahead and change this to always allow. That means anytime the AI thinks that it needs to talk to Home Assistant, it can use whatever tool it wants. But, of course, I wouldn't recommend doing that. You do

13:51

Speaker A

have granular control. So, you can go in here and turn off anything that you do not want it to have access to. You can click the block button and immediately it takes that tool away from the AI and

14:01

Speaker A

it can't use it until you give it permissions. So, really read down through this list and note what things you're comfortable with. And if you're just getting started, I'd recommend going ahead and turning off the write delete tools and just leaving it in

14:13

Speaker A

readonly mode. That way you can have full access and it can analyze and tell you what your problem is. But then it's up to you to make those changes. So for right now, I'm going to go ahead and

14:21

Speaker A

leave it as it is. And again, at the top, one other thing to keep in mind is it does show in plain text that URL. So if you're showing this off to friends or you're going to do a screen share, make

14:30

Speaker A

sure you blur that out. So you'll notice here it says web connection, which means it doesn't require the desktop app. So, you can actually flip over to the Claude mobile app or the Claude web app and be able to interact with your smart home

14:44

Speaker A

all through there as well. So, as a quick test, let's go ahead and hop back in to Claude. I'm going to go ahead and just click under projects. I've got a Home Assistant one here, and we'll just change to something that uses less

14:56

Speaker A

tokens. and just say, "Hey, as a test, can you let me know how many Z-Wave devices I have in my smart home?" So now, since this is the first time it's going to use this tool, it's popped up

15:13

Speaker A

and it's asked me, "Hey, do you have want to give me permission to use a git device to be able to go in and see what these are?" And of course, if you want to do this only one time, you can click

15:23

Speaker A

the arrow and say allow once. But of course, I've used this quite a bit before. So, I'm going to go ahead and say always allow. So, now it's going to go in, shake hands with Home Assistant, and start pulling that information that

15:33

Speaker A

it needs through that Home Assistant connector and then display it here. So, right there, that quickly, now it tells me I have 28 Z-Wave devices, which is accurate. It also talks about what the controller is and all the different end

15:44

Speaker A

devices and lists them out in detail. So, that's pretty cool and actually very, very fast for it to respond that way. All right, so let's go ahead and quickly switch over to Chat GPT and I'll show you how to get it set up in there.

15:57

Speaker A

Couple things to note with Chat GPT, you're going to need to at least have either the Pro or Plus. So that's sort of the equivalent of the Pro or the Max plan over on Claude. That gives you access to the developer tools so you can

16:08

Speaker A

add in the connector. And as of right now, end of July 2026, it's in that developer mode is where you add this.

16:15

Speaker A

But unfortunately, CHPQ is moving stuff around. So, uh, it might be different for you. So, you might just want to ask Chat GPT where it is. So, I find this easier for Chat GPT to do it via the web

16:27

Speaker A

version, not the desktop version, cuz things keep moving around. So, we're on the Chat GPT website. We're signed in.

16:33

Speaker A

We'll go down to my username. Click on settings. Then, we'll go over to plugins and scroll down to the bottom and click developer mode. Now, we want to activate developer mode. This way we can add our own custom configurations.

16:49

Speaker A

So now that we've got developer mode activated, we just need to add a new connector. So we'll go click on our profile, back to settings, go down to plugins, scroll to the bottom, and click browse plugins. Then from here at the

17:03

Speaker A

top right, we're going to click the plus button, which is going to allow us to create a new plugin. So we're going to call it home assistant.

17:10

Speaker A

And then we want to click under connection. We're going to paste in that big URL again. So, we're going to select authentication, no off, and check the box that says I understand and want to continue and click create. Of course,

17:22

Speaker A

you can create your own custom icon, which I'd go back and do that some other time. Um, then we need to click sign in with Home Assistant.

17:30

Speaker A

And once that's finished, then you now have access to that tool. One thing to note though in chat GBT is sometimes you actually have to explicitly say use the home assistant connection tool. So, you might have to call it by name. Claw

17:41

Speaker A

doesn't seem to have that issue. So, as a real quick test, let's go ahead and flip over to Chat GPT and we're going to ask a simple question. How many Ziggby devices do I have in my home assistant?

17:50

Speaker A

You can see here, you don't even need to give it the extra context of telling it to use the MCP connector. It automatically knows and it returns that I have 68 actual devices and 69 devices associated with it, meaning one's dead.

18:05

Speaker A

So, that right there shows us that the connection is open and working just fine. So before we jump into setting up and configuring our LLMs, let's quickly look at some of the security settings that we can set in the HAMCP

18:18

Speaker A

connector. So once it's installed over here on the sidebar, we have the HAMCP. We can click right on that and it'll take us directly into the configuration for the settings.

18:30

Speaker A

Now there are a ton of options in here. One, you can enable readon mode where it'll disable all the right tools.

18:36

Speaker A

That's a good way to get started. You can go in and disable individual tools.

18:42

Speaker A

So none of the LLMs can access them. But the main thing we're going to go to here right now is the entity visibility. So if we check this box here, we have a few options that we can use to disable its

18:54

Speaker A

access to certain types of entities. So one of the first things you can do is you can check this in force mode, which is going to forcibly hide anything that you've selected in these filters below.

19:06

Speaker A

What that does is it hides it even from a direct action. Whereas by default, it says up here, what it's going to do is basically when it searches, it's not going to return it if you have this checked. But if you want to disable all

19:18

Speaker A

access to specific items, you need to check this to where they're hidden and unreachable across all the tools. So right here, you can go in and disable certain areas or certain labels, which we'll look at creating a label here. You

19:30

Speaker A

can also go down here and type in specific entities, and then that will block those entities. Now, this has to be individual entities. So, if you have, say, like the Tesla, there's something like 70 entities. So, you'll need to go

19:40

Speaker A

in here and add every single item. So, I think it's a lot easier to go up here and actually create a label and have that entire device blocked. Then down here, we can do a inverted where we can

19:50

Speaker A

actually restrict it to an allow list, which is this is the only things I wanted to have access to are certain areas or certain labels or certain entity IDs. And finally, at the bottom here, we can actually have it follow the

20:01

Speaker A

same assist exposure. So if you using the assist engine, you can go in here and actually match it to what items that are only being visible to assist. So to do that, let me hit save real quick. So

20:16

Speaker A

if you go to any entity ID, so we're going to bring up say my shades here. If I click on the gear icon, go down here to voice assistance, it would then follow if you have assist activated, it

20:28

Speaker A

would then allow it to be accessible with HAMCP. So, what I'm going to do is real quickly head over to my entity list and I'm going to type in lock dot. This will show all the locks that are currently in my house and I'm going

20:44

Speaker A

to click on each one of them. Go up to the gear icon and then click add label. So, we're going to create a new label. We're going to call it no MCP.

20:58

Speaker A

So, I'm going do that for the rest of my locks. And I'm going also going to do my garage doors.

21:13

Speaker A

Now we'll head back to HCP, go back to entity visibility, and we want to do under labels, we'll say no MCP as one of the labels and hit save. Now the MCP server should not be able to access anything with the label

21:26

Speaker A

no MCP. So those shouldn't even exist as far as they're concerned. So now before any of these changes will actually take effect, we have to go in and restart the custom integration.

21:37

Speaker A

So the best way to do that is to go into your integrations, go find HMCP, click on the three dots, and then click disable. Wait about 10 seconds, and then click enable, which will reactivate it.

21:52

Speaker A

And then we're going to switch over to Chat JPT where I've done a quick test.

21:57

Speaker A

Now, I've already asked it to see if it could locate my locks, which it was able to find before I restarted the integration. Now, I've asked it, can you give me some specific details on the front door lock? And we'll see because

22:11

Speaker A

it's already in memory, it knows the entity ID, but when it tries to check out any detailed information, it's being blocked by the MCP integration.

22:19

Speaker A

So, now you can go through and add in what specific entities you want to block. You need to do these one at a time or again you can create that allow list where it'll only access certain entities if that's the way you want to

22:31

Speaker A

handle it. So now that we've got both of these LLM set up, let's go ahead and see an couple quick examples of how to use them. And for that I'm going to switch back to Claude. Let's do, hey, can you

22:43

Speaker A

create a an automation that sends a notification to my phone, either S24 Ultra, the Samsung S24 Ultra, using the Home Assistant app, anytime one of my Z-Wave devices goes offline, boom.

23:10

Speaker A

Go. So now, anytime that it goes in to try to access a tool that it hasn't accessed before, it's going to have to ask my permission. So it's going to take a few minutes here to build out what it

23:20

Speaker A

thinks it's going to do. And so now it's going to ask me if I want to give it permission to get Home Assistant best practices and skill guide from Home Assistant. Of course, I want to do that.

23:29

Speaker A

Now, as we can see here, doing interacting with an LLM this way is very much back and forth. So I'm asking to do something, it's doing something. It's not automatic. When we take a look at my Alfred bot here in a minute, we'll talk

23:41

Speaker A

about how custom instructions become very important and how we can use that to scope down what specific devices or range of devices we don't want it to talk to. So, it's popped up. It's going to want to use the search tool. So,

23:52

Speaker A

we're going to go ahead and allow that. So, you're going to have to sit here and kind of watch and go back and forth with the AI in order to determine what tools it's going to need access to and if

24:01

Speaker A

you're comfortable with it doing that specific item. So now that Claude's actually processed everything, it's going to go ahead and create that automation. So we're going to go ahead and say and this time say allow once again if you don't want to give it full

24:14

Speaker A

range to do whatever it wants to do. So it's gone ahead and created the new automation and flagged one thing of note that the notify service gets a little confusing. So I need to go through and validate that that actually worked,

24:26

Speaker A

which we'll check out here in a second. And then it wants to do a list to kind of validate everything.

24:31

Speaker A

So, it's going to continue working here in the background trying to figure out where that notify service actually is.

24:38

Speaker A

All right. So, now Claude has finished going through and doing its analysis and it said, "Hey, I found the right notify entity and it's corrected everything." So, then it kind of tells us a quick summary what's in place. The

24:52

Speaker A

automation's created called notification Z-Wave device offline and it wants us to go in and validate that it's the right target. This is that same issue I mentioned before with my AI agent that it actually didn't have the right name

25:06

Speaker A

in the automation because I changed it on accident. So, we're going to switch back over here to Home Assistant. I'll go down to settings, go to automations, and then at the top here, I'm going to go ahead and type in, you know, Z-Wave.

25:18

Speaker A

We'll see what comes up. So, we have this notification Z-Wave devices offline. Click on that. And at the top here, we've got a trigger of all the devices. So, it went through and it found every single one of my Z-Wave

25:30

Speaker A

devices and found its associated status entity. So, it's got all of those in there, which you can kind of see at a quick glance, things like items that are asleep versus being being awake and having a heartbeat. You can see a couple

25:43

Speaker A

of these that are showing dead. So, those those should show me an error. That's actually my keypad right out here. And I think it ran out of batteries.

25:51

Speaker A

So, if any of those are triggered, then it's going to simply perform an action of sending me a notification. But you can see here it had an error. It called out saying, "I don't think that's right, so you should investigate." So, if we

26:03

Speaker A

scroll down to the bottom here, it kind of summarizes what it's done. It went ahead and applied everything. It tried to follow the instructions that I gave it, and it came across an error.

26:12

Speaker A

Couldn't find the right entity ID and said, "Hey, I did a quick search. I've already created the automation for you, but I went ahead and searched. I think I found the right one. What do you want me to do next?" So now it says apply it

26:25

Speaker A

suggested fix which is that's the correct entity ID or I could leave it as it is and fix it myself. So I'm going to say let's go with option one.

26:35

Speaker A

So it's going to call that creator update automation tool again. So go ahead and allow it. All right. So it says there's no more validation warnings. So let's click back over here.

26:44

Speaker A

We'll refresh the page. Go back in. And sure enough now it has an action. It has the right message entity ID and the title all put in there ready to go. So now the next time one of these devices goes offline, I'll get a

26:58

Speaker A

nice notification on my phone telling me exactly which device is offline and I can go deal with it as I need to. It even wrote a nice little description of what the automation does. So that's a nice touch so it's easy to remember what

27:11

Speaker A

the heck this is sometime in the future. Of course, since I've given it full control, I could ask Claude to turn off any light I want in the house and it would do something similar as if you used a Google Home.

27:22

Speaker A

So, that's great, especially if you're going to be building something brand new and you want to work with an AI to develop some sort of automation. But what about that automated repair loop that I showed you earlier on? So, now

27:33

Speaker A

we're going to hop into my automated agent, which is named Alfred. So now we're going to take a look at kind of the last bit which is what I teased at the beginning that automated loop that I've got created with my AI agents. Now

27:45

Speaker A

I use an AI agent system called Hermes agent and I really don't have time to get into that for today's video but effectively it uses an LLM like Claude or Chat GPT and then adds a bunch of extra context on top of it so you can

28:00

Speaker A

continue to interact with it and it stays smart and remembers a lot of different things. There are a whole bunch of great videos out there on YouTube on setting up Hermes agent.

28:08

Speaker A

That's the one that I selected. And in all reality, it actually is powered by Chat GPT. I just sign in and it uses my chat GPT allotted tokens, but there's a ton of flexibility out there. So in there, you can create your own agents.

28:23

Speaker A

You can have a main agent, you can create sub agents, all that sort of stuff. So, I went ahead and asked my main agent to generate and create me a specific sub agent named Alfred that can take care of all of my home assistant

28:34

Speaker A

interaction. So, I've got some examples in the blog post of some of the suggested, you know, guard rails or things that you can use to add in for that agent information. I kind of took all my personal information out of it,

28:45

Speaker A

but feel free to use those to get you started. So, I've already set him up once before.

28:51

Speaker A

Um, I'm going to kind of mimic that now because of course I've changed that URL because I created kind of the new method to access it. So, I'm going to kind of give you a prompt that you can use to

29:02

Speaker A

set it up. Of course, you can just ask your AI agent, how do I set this up? Um, but I'm just going to basically talk to it like I would a regular regular person.

29:11

Speaker A

Hey Alfred I want to give you access to my home assistant instance. via the H hmcp server.

29:29

Speaker A

Here is the URL for that server. Please get connected and let me know what information you need.

29:46

Speaker A

And of course, I'm giving him a local IP address because this runs on a on a server I have down in the basement. The agent's processes and knowledge run on my basement server, but the thinking still happens in OpenAI's cloud. So,

29:59

Speaker A

since it is local, I'm going to use that direct access link. Now, of course, the right way to do this would actually be to have it set up everything and then you go into a configuration file and pop

30:09

Speaker A

that URL in. That way, it doesn't get sent back to the LLM. Um, but for ease of use, I'm just going to have the AI take care of it all for me right now.

30:19

Speaker A

And then later on, I can go back and replace that with a new version of it and respin all of those keys. So now it's going to go in and check out that MCP server and start building out

30:32

Speaker A

whatever is required and necessary to scaffold it so it can actually talk to the server.

30:39

Speaker A

One of the nice things I like about Hermes agent is that I can actually watch as it goes along and it's telling me what it's doing, what commands it's it's what commands that it's actually executing. So, I've set up

30:52

Speaker A

my Hermes agent where I can interact with it via Telegram. So, that's why you see me in Telegram talking through with it there. You can, of course, set up whatever you want, but I prefer Telegram cuz it's I only use it for this. If you

31:04

Speaker A

want a full breakdown of setting up something like this or open claw, let me know down below. I could possibly put something together even though I'm not a subject matter expert on it. Went ahead and changed the tool list. Didn't make any and said

31:19

Speaker A

specifically didn't make any changes to home assistant, but went ahead and set up the endpoint. So big green check mark connection works fine. Current state reporting back what's happening because again I'm replacing an existing one. You won't see a lot of this stuff. So I need

31:32

Speaker A

to run the commit. It's telling me I need to run the command reload MCP which will go ahead and kind of refresh its database of MCP servers. And then a lot of these have a prompt similar to what

31:42

Speaker A

we saw before where I have to approve it. Now, of course, I've been interacting with this um agent quite a bit. So, he even mentioned that he's going to take all the approvals that I've already given him and retain that

31:54

Speaker A

so he can go ahead and use those tools as necessary. So, everything's reloaded. So, as a quick test, let's go ahead and say, "Hey, Alfred, can you turn off the neon light in my office?" So, as you can see, the light behind me

32:15

Speaker A

just turned off. Obviously, it's not as fast as it would be if I use the Google Home, but of course, it's got a lot more at its disposal. I could tell him to turn off a bunch of lights, turn them

32:25

Speaker A

on, whatever you can do to Home Assistant, you can interact with it and have it done via the the text chat.

32:33

Speaker A

So great. Can you turn it back on? So there we go. It's back on now. So great. This is all well and good. We can control lights. You can do all the same stuff we did with Claude earlier where

32:46

Speaker A

we went in and had it build automations. But let's get into that control loop that we talked about before. And that's what a concept if you're not big into computers. It's what's something called a cron job. A cron job is just a job

32:58

Speaker A

that runs at a specific time of day. And I've already programmed Alfred to run that cron job, but I've got the full prompt on how to set it up in the description. But let's take a look at what the cron job actually says.

33:10

Speaker A

Can you give me details on the daily selfhealing cron job? So to create any s type of scheduled task with your agent, you can just tell the agent to do this every day at 9:00 a.m. or every weekday or every other

33:32

Speaker A

day. You can just tell it in plain text and it creates the automated job for you. So now it's going to reach back into its config and pull out all the information about that cron job. So you can see kind of how I set it up. All

33:45

Speaker A

right. So we went ahead and pulled in all the details. It's got its own hash ID, the name of it, when it runs. So every day at 9:00 a.m., the next time it's going to run, the last time it ran,

33:54

Speaker A

the status of that one, uh what it does, it sends a command back into this Telegram chat. Goes on forever, and what skills it utilizes. So here the command I've asked for it is it's designed to scan the home assistant automation

34:08

Speaker A

failures goes into my error level logs filters out anything with automation ignores non-automation issues and then classifies them into different scopes.

34:19

Speaker A

Is this like a one-time issue? Oh, the battery's dead or has this been erroring numerous times over and over again? For one of the high confidence issues, probably the highest priority one, it goes and does a real deep dive into it

34:29

Speaker A

and then sends me back that proposal and then stops. Doesn't do anything. it does not have the ability to make any changes as per this set of rules. And then I have a whole list of safety rules and

34:41

Speaker A

this is how I go in and scope out what it does and does not have access to. So it says that you are explicitly prohibited from making changes on its own. It must not modify anything until I give it explicit approval. And some hard

34:53

Speaker A

exclusions are no locks, no garage door automations or garage cover targets, no alarm security automation changes, nothing related to my Tesla, no guest values, one proposed change at a time, create a snapshot before it writes, and all everything must match. So meaning

35:12

Speaker A

it's got to have the same hash. There's no errors or issues with it. This is the last time it ran successfully, and it told me what it did. So, it went ahead and identified what we talked about, which is that error that I talked about

35:23

Speaker A

at the top of the video where the alert wasn't getting to my phone. And it's the same problem we saw with the automation we created. It was targeting an old entity ID. So, it's proposed change is to switch out the mobile app for S24 for

35:38

Speaker A

the new correct one that we've already identified. So, then all I have to do when I get that notification on my phone is just say, "Awesome, go ahead and apply that change." It'll do the fix in the background, report back exactly what

35:49

Speaker A

it did, and there it goes. Now I've got a repaired automation that starts working the next time that it's triggered. And I also have it where it gives me kind of one big one that it can recommend solving, but then it usually

36:01

Speaker A

recommends two or three other fixes each day. So hopefully after weeks and weeks of this, most of your automations have been fixed. And of course, you can change this and diversify it to do whatever you want it to do. You can have

36:11

Speaker A

it identify offline devices, devices that are low on battery, anything. You can build that into these cron jobs and have it automatically generate it and and take action on it even with your approval. So, back to the question that

36:24

Speaker A

this whole video is based on. Would I give an AI the keys to my smart home?

36:29

Speaker A

And my honest answer is yes. Of course, I've already set it up, but with only some of the keys and with my hand on the brake, Alfred can read my logs and suggest a fix to the broken automation

36:40

Speaker A

while I'm asleep. But it can't do anything until I give it permission. It can't touch my locks, my garage, or anything connected to my car. It's just programmed where it can't have access to those things. So, in my opinion, if you

36:52

Speaker A

follow these two rules, then having an AI in your home is genuinely a helpful tool instead of a large liability.

36:59

Speaker A

ignore them and you might get something unexpected or even worse, giving a confident stranger the run of your home.

37:05

Speaker A

So, if you watch this entire video and decide the trade-off really isn't worth it, then I've done my job. There's no wrong answer here, only an informed one, and that's my goal. Now, of course, everything I showed you today runs

37:16

Speaker A

through a cloud AI, Alfred included. He actually runs on GPT 5.5 via the Hermes agent. And of course using a cloud AI that means every log that he reads and every fix that he proposes takes a round trip through open AI. I don't want

37:34

Speaker A

somebody else's cloud reading my house. But the answer to that is to run the whole thing on a local model instead.

37:39

Speaker A

Nothing leaving your network. A local model that's capable enough to diagnose automations reliably needs real hardware with a large context window. We're talking a serious GPU investment, not a spare Raspberry Pi. And in today's world, I'm not willing to make that

37:56

Speaker A

investment. What it actually takes to build a self-hosted smart home AI, the hardware, the model, the costs, and whether it's worth it versus just accepting the cloud trade-offs. So, let me know down in the comments if that's a

38:08

Speaker A

video that you want to see. But you don't need any of that to get started.

38:12

Speaker A

So, if you take one thing from today, you can connect it as read only and just ask your house, why isn't this working?

38:18

Speaker A

Don't give it any ability to take action. So that way next time that something that breaks, you can still have it walk you through and then you are in control of what changes are made.

38:27

Speaker A

It's not as convenient, but it's a lot more secure. That's how I operated before I found out about MCP servers. So if you're just getting started with Home Assistant, I've got a great beginner playlist right over here. Thank you

38:38

Speaker A

again for making it this far and I'll see you on the next

Topics: Home Assistant Smart Home AI Automation MCP Server HAMCP Alfred AI Agent Home Automation Security Self-Healing Automation Claude AI ChatGPT

Study this video

- [Flashcards from this video](https://sozai.app/tools/ai-flashcard-generator/?from=transcript&id=78242)
- [Quiz on this video](https://sozai.app/tools/quiz-generator/?from=transcript&id=78242)

Free, made by AI from this transcript. No signup.


---
This is the markdown twin of https://sozai.app/transcript/ai-control-smart-home-should-you/ — the same content, without the markup.
Published by SozAI (https://sozai.app). Reuse and quotation are allowed with attribution and a link back.
Machine-readable index: https://sozai.app/llms.txt · data API: https://sozai.app/api/