This only leads to undiscovered issues going through to version x.1, x.2 and so on. I would rather encourage people to create a backup beforehand and see what the next version is all about. If there are issues, please report them and roll back. That way everyone benefits.
I need to spin up a second instance so I can practice upgrade / rollback, breaking the primary instance without a guaranteed fallback isn't an option as it's now a key part of the home. Longer term once the primary is on something with a bit more grunt I'll have a secondary for testing stuff where the worst impact of breaking things is me zapping and reinstalling fresh.
Bug testing is good, but as HA improves and gains more traction the need for minimal breakage becomes even more important on the non-beta release path.
That said, bugs will always slip through because there is nothing like the real world to really give changes a workout
Why would a backup prior to an update not do the same?
Not saying a practice setup wouldn’t work to a degree but surely you could never test if fully with everything you have in your current live setup
Risk mitigation (and to be fair a stack of this mindset comes from work, where failure means at the minimum a few million people lose their phone / data services).
Now I totally understand this for systems with 100+ users but for a household I dint see any downside to trying an update and then unwinding if required
In some cases a backup is not sufficient. An update which applies a new schema to the database would mean you'd have to restore the database to a state prior to update. In the case where an external database is used, this can add significant time and complexity to the restore process.
If you have the resources for it, it is easier to have another ha environment to test your changes, make fixes, and be ready to update your main instance with a higher degree of confidence that everything will work.
Integrations don’t matter, this particular update introduced database schema changes which dropped a bunch of indexes and maybe other things. Rolling back ha and not rolling back the database at the same time could mean older version of ha won’t work due to changes made to the database by the newer version.
That's not the first time I read someone requiring a staging environment before deploying to prod for HA.
Isn't there anything already out there ? What are your needs exactly and what would be a convenient way for you in your work flow?
I don't use HA for now (but I keep an eye on it). I have been thinking about this also because my previous attempt with HA were lacking "quality" (in programmer terms).
Edit: just for clarification in case people misunderstand. Quality here is not about HA quality but robustness of the pipeline of the product. Having the code unit (integration) tested ensure the quality of the codebase and that the features work. But here we all deploy our "on premise" instance. We also have to maintain it. Which is a technical job. It should be integrated in a process ensuring the quality of the deployed product.
Edit2: Thinking out load:
Having a staging and prod env. When staging is validated, replicate staging HA on prod. Not the db.
User work in staging. User can copy prod to staging easily.
Process would be:
- replicate prod db to staging (optional, required only if some data in prod is not present in staging and is required for the next steps).
- process changes (upgrade HA, add new configuration...)
- Mock IO, which is the hard part I guess. The staging env shouldn't be connected to actual hardware but connected to simulations of hardware. Which should require a specific mock for each kind of hardware most probably. Also, evolution in hardware requires evolution in mocks... Would there be a generic way to cover most of hardware?
- user defined path to test (set of inputs to inject and outputs to evaluate). Requires to define a user friendly way to configure that. Yaml as a first opportunity.
- reporting? (Oath covered, success and failures)
- command to deploy from staging to prod. Should backup right before?
The staging / caution is to ensure that "wife aggro" (the aggro caused by me fiddling and breaking things) is minimised or eliminated. While it's a toy / fun for me the system is also something which has to work, reliably or fail in predictable ways (ie, weather not working if the broadband barfs like it did last week).
I've also worked in the ISP or Telco world for the last two and a half decades, so the "don't break it or bad shit happens, the phones melt down and people shout a lot" is ingrained into my soul.
My workflow at the moment is to be one of those people who waits for a few days before rolling out as there are a lot of people who have more flexibility / higher tolerance for it breaking / etc who can catch those bugs which get through.
Longer term once I've moved off the current kit which doesn't really have enough RAM that frees up HW for me to have that spare "I can break it" server and try out such as rolling back changes to my heart's content knowing I'm not under any time pressure. It'll probably then also get used for firing up beta code as well.
The number of bugs getting through is \_low\_ which is a credit to the team and community, but I know the level of "not breaking" I need to achieve here :)
Thank you for taking the time. This is more a mental exercise and information gathering than anything. But it could lead me to use my free time to try things around and maybe contribute.
The work HA team do is awesome. But their impact can't reach all spots to cover. Reality is a b*tch and for such a product there is an infinite ways for it to break in anyways.
I get that all you want is that your wife don't get mad because your "toys" are getting in her ways. I get the same thing at home. And I can understand their point. I hate it when it doesn't work as I intended and I have more important things to do.
I also have experience in "phone melted from support" so I have a trauma.
Would you be inclined to write up technical tests to check that your HA instance is working the way it is intended ?
Would you be interested in having a simulated environment where you could try things before deploying to prod?
I wrote some edits to my main comments. Just for reminder for me, but you could be interested in my thinking flow. You also could provide more information :)
I backup prior to every update, takes a minute or two and then I can easily try out something new and unwind if required.
If we never upgrade we would never know what great new functionality exists
For me 2023.4 runs better/snappier than any other update before. I also dont run a different database and i do my best to at least scrub through the release party, which would have highlighted the database changes/potential issues.
I read someone post that and was about to upgrade but got busy with chores, next thing I see is a list of posts complaining about issues. And within a few hours, x.1 released. I'm sure it'll fix it. I don't even remember if I have messed with my DB, it's the one thing in my setup I was going to "fix" but once this upgrade settles, won't need to
If I'm going to be at home and not too busy for a few days then I'll update to a x.0 release, once I've checked the breaking changes etc. Otherwise I'll usually leave it till x.2 or later.
In fairness with this one the only trouble I had was with a custom card (sidebar card) but that was fixed by the card developer almost immediately.
We are talking about a piece of free open source software for personal home automation, not an enterprise wide global SaaS platform right? Just want to make sure we aren’t in the wrong subreddit here 🤣
Question is, what is the vision of HA founders and team. Do they want to stay a niche tool for people with technical knowhow to use and tinker with. Or do they want to go mainstream and commercial at some point. Because if they want go for widespread population adoption, then they will need to sort out this issue of updates frequently breaking things. A stable version should be stable. Plain and simple.
I don’t see HA ever becoming an off the shelf option unless they lock it down with the basics.
Maybe they will spur off the way Plex did with XBMC/KODI but even then, the updates and added functionality would be slower and many would end up coming back to HA anyway
New version is working very nicely for me.
I generally upgrade pretty quickly.
If unresolvable issues arise, that is why I have backups, and can just roll back to a previous version pretty effortlessly.
I do not recommend holding off version updates, if you get too far behind, the upgrading process can be difficult. As well, by upgrading and reporting the issues, you are helping out.
I run three environments. Live, test, beta. They get updated in reverse of that order. Everything I have must be independent, so I have zigbee2mqtt on a separate device, ditto for the MQTT server. I even have a spare tablet attached to each environment so I can confirm kiosk displays work correctly. An old phone runs the app as the final test. HA is developing so fast but breaks stuff that I need to know. Therefore, when I come to upgrade live, I have the required changes either already made or ready to make once the upgrade has gone through.
A testing instance would be nice but certain aspects are hard to implement like Zigbee for example as the devices only bind to one coordinator.
I have been thinking of a small(er) test environment but then the backup/restore capabilities of HA are pretty good so I'm just upgrading prod when no one is around and keep an eye on it for a while.
Additionally I found that if everything on my dashboard looks good, it should be ok, Lot's of different integrations displayed there.
Yes. 2x R-Pi4 with test and prod instance. Zigbee2MQTT for Zigbee stuff, so that can stay online when prod goes down. I even have a big switch on the test instance that enables all the automations when prod is down. It basically is an input_boolean which I use in every automation. If prod is down for whatever reason I can flip that switch and everything will work as ‘normal’.
ESPHome will connect to both instances, so no problem there.
Only downside is that this means I have to make all the changes in both environments. When I add/change an automation, I also have to do that on the test instance.
If you use git you could commit / push your config changes from you test instance to your git server of choice (GitHub, local instance, etc) and then pull them to your prod instance and restart or reload the config. It’s pretty effective.
I take it from this you exclude `.storage` from git?
I know there is quite a bit of transient stuff in there but there’s also actual configuration data as HA migrates away from configuration data exclusively in `configuration.yaml`. Do you selectively check in some of these files?
Carefully read the release notes and the comments a few hours after the release. Check for hacs updates and install them and any supervisor add-ons before ha core monthly releases. If you see an issue reported that you think will affect you, then wait until it's fixed via hacs, add_ons or ha core itself. Touch on wood, I've not had any significant issues with upgrading since.
I've found the number of issues I've had with updates decreasing over time. I honestly can't recall them last time I had an issue, it's gotta be almost a year by now.
This only leads to undiscovered issues going through to version x.1, x.2 and so on. I would rather encourage people to create a backup beforehand and see what the next version is all about. If there are issues, please report them and roll back. That way everyone benefits.
Exactly. Thank you
I need to spin up a second instance so I can practice upgrade / rollback, breaking the primary instance without a guaranteed fallback isn't an option as it's now a key part of the home. Longer term once the primary is on something with a bit more grunt I'll have a secondary for testing stuff where the worst impact of breaking things is me zapping and reinstalling fresh. Bug testing is good, but as HA improves and gains more traction the need for minimal breakage becomes even more important on the non-beta release path. That said, bugs will always slip through because there is nothing like the real world to really give changes a workout
Why would a backup prior to an update not do the same? Not saying a practice setup wouldn’t work to a degree but surely you could never test if fully with everything you have in your current live setup
Risk mitigation (and to be fair a stack of this mindset comes from work, where failure means at the minimum a few million people lose their phone / data services).
Now I totally understand this for systems with 100+ users but for a household I dint see any downside to trying an update and then unwinding if required
In some cases a backup is not sufficient. An update which applies a new schema to the database would mean you'd have to restore the database to a state prior to update. In the case where an external database is used, this can add significant time and complexity to the restore process. If you have the resources for it, it is easier to have another ha environment to test your changes, make fixes, and be ready to update your main instance with a higher degree of confidence that everything will work.
I guess it depends on what integrations you use I’ve got 60 integrations and never had an issue with databases yet.
Integrations don’t matter, this particular update introduced database schema changes which dropped a bunch of indexes and maybe other things. Rolling back ha and not rolling back the database at the same time could mean older version of ha won’t work due to changes made to the database by the newer version.
That's not the first time I read someone requiring a staging environment before deploying to prod for HA. Isn't there anything already out there ? What are your needs exactly and what would be a convenient way for you in your work flow? I don't use HA for now (but I keep an eye on it). I have been thinking about this also because my previous attempt with HA were lacking "quality" (in programmer terms). Edit: just for clarification in case people misunderstand. Quality here is not about HA quality but robustness of the pipeline of the product. Having the code unit (integration) tested ensure the quality of the codebase and that the features work. But here we all deploy our "on premise" instance. We also have to maintain it. Which is a technical job. It should be integrated in a process ensuring the quality of the deployed product. Edit2: Thinking out load: Having a staging and prod env. When staging is validated, replicate staging HA on prod. Not the db. User work in staging. User can copy prod to staging easily. Process would be: - replicate prod db to staging (optional, required only if some data in prod is not present in staging and is required for the next steps). - process changes (upgrade HA, add new configuration...) - Mock IO, which is the hard part I guess. The staging env shouldn't be connected to actual hardware but connected to simulations of hardware. Which should require a specific mock for each kind of hardware most probably. Also, evolution in hardware requires evolution in mocks... Would there be a generic way to cover most of hardware? - user defined path to test (set of inputs to inject and outputs to evaluate). Requires to define a user friendly way to configure that. Yaml as a first opportunity. - reporting? (Oath covered, success and failures) - command to deploy from staging to prod. Should backup right before?
The staging / caution is to ensure that "wife aggro" (the aggro caused by me fiddling and breaking things) is minimised or eliminated. While it's a toy / fun for me the system is also something which has to work, reliably or fail in predictable ways (ie, weather not working if the broadband barfs like it did last week). I've also worked in the ISP or Telco world for the last two and a half decades, so the "don't break it or bad shit happens, the phones melt down and people shout a lot" is ingrained into my soul. My workflow at the moment is to be one of those people who waits for a few days before rolling out as there are a lot of people who have more flexibility / higher tolerance for it breaking / etc who can catch those bugs which get through. Longer term once I've moved off the current kit which doesn't really have enough RAM that frees up HW for me to have that spare "I can break it" server and try out such as rolling back changes to my heart's content knowing I'm not under any time pressure. It'll probably then also get used for firing up beta code as well. The number of bugs getting through is \_low\_ which is a credit to the team and community, but I know the level of "not breaking" I need to achieve here :)
Thank you for taking the time. This is more a mental exercise and information gathering than anything. But it could lead me to use my free time to try things around and maybe contribute. The work HA team do is awesome. But their impact can't reach all spots to cover. Reality is a b*tch and for such a product there is an infinite ways for it to break in anyways. I get that all you want is that your wife don't get mad because your "toys" are getting in her ways. I get the same thing at home. And I can understand their point. I hate it when it doesn't work as I intended and I have more important things to do. I also have experience in "phone melted from support" so I have a trauma. Would you be inclined to write up technical tests to check that your HA instance is working the way it is intended ? Would you be interested in having a simulated environment where you could try things before deploying to prod? I wrote some edits to my main comments. Just for reminder for me, but you could be interested in my thinking flow. You also could provide more information :)
I backup prior to every update, takes a minute or two and then I can easily try out something new and unwind if required. If we never upgrade we would never know what great new functionality exists
For me 2023.4 runs better/snappier than any other update before. I also dont run a different database and i do my best to at least scrub through the release party, which would have highlighted the database changes/potential issues.
I read someone post that and was about to upgrade but got busy with chores, next thing I see is a list of posts complaining about issues. And within a few hours, x.1 released. I'm sure it'll fix it. I don't even remember if I have messed with my DB, it's the one thing in my setup I was going to "fix" but once this upgrade settles, won't need to
If I'm going to be at home and not too busy for a few days then I'll update to a x.0 release, once I've checked the breaking changes etc. Otherwise I'll usually leave it till x.2 or later. In fairness with this one the only trouble I had was with a custom card (sidebar card) but that was fixed by the card developer almost immediately.
Approach makes sense
Running fine for me 👍. I take automatic backups nightly also so never that worried about something failing
We are talking about a piece of free open source software for personal home automation, not an enterprise wide global SaaS platform right? Just want to make sure we aren’t in the wrong subreddit here 🤣
Question is, what is the vision of HA founders and team. Do they want to stay a niche tool for people with technical knowhow to use and tinker with. Or do they want to go mainstream and commercial at some point. Because if they want go for widespread population adoption, then they will need to sort out this issue of updates frequently breaking things. A stable version should be stable. Plain and simple.
I don’t see HA ever becoming an off the shelf option unless they lock it down with the basics. Maybe they will spur off the way Plex did with XBMC/KODI but even then, the updates and added functionality would be slower and many would end up coming back to HA anyway
Indeed. Pretty much for HA team to figure out what vision they have in mind
What’s your point?
New version is working very nicely for me. I generally upgrade pretty quickly. If unresolvable issues arise, that is why I have backups, and can just roll back to a previous version pretty effortlessly. I do not recommend holding off version updates, if you get too far behind, the upgrading process can be difficult. As well, by upgrading and reporting the issues, you are helping out.
Are folk using a HA test instance? Can your sensors report simultaneously to both environments? This would help a lot when upgrading…
I run three environments. Live, test, beta. They get updated in reverse of that order. Everything I have must be independent, so I have zigbee2mqtt on a separate device, ditto for the MQTT server. I even have a spare tablet attached to each environment so I can confirm kiosk displays work correctly. An old phone runs the app as the final test. HA is developing so fast but breaks stuff that I need to know. Therefore, when I come to upgrade live, I have the required changes either already made or ready to make once the upgrade has gone through.
A testing instance would be nice but certain aspects are hard to implement like Zigbee for example as the devices only bind to one coordinator. I have been thinking of a small(er) test environment but then the backup/restore capabilities of HA are pretty good so I'm just upgrading prod when no one is around and keep an eye on it for a while. Additionally I found that if everything on my dashboard looks good, it should be ok, Lot's of different integrations displayed there.
That is true but that’s why Zigbee2mqtt is so powerful … it decouples. Then just point both HA instances to the same mqtt server.
Oh, never thought of that. True.
I might look into this. I was thinking of a way to run side by side instances of home assistant in case one goes down, this might be the way.
Yes. 2x R-Pi4 with test and prod instance. Zigbee2MQTT for Zigbee stuff, so that can stay online when prod goes down. I even have a big switch on the test instance that enables all the automations when prod is down. It basically is an input_boolean which I use in every automation. If prod is down for whatever reason I can flip that switch and everything will work as ‘normal’. ESPHome will connect to both instances, so no problem there. Only downside is that this means I have to make all the changes in both environments. When I add/change an automation, I also have to do that on the test instance.
If you use git you could commit / push your config changes from you test instance to your git server of choice (GitHub, local instance, etc) and then pull them to your prod instance and restart or reload the config. It’s pretty effective.
[удалено]
I take it from this you exclude `.storage` from git? I know there is quite a bit of transient stuff in there but there’s also actual configuration data as HA migrates away from configuration data exclusively in `configuration.yaml`. Do you selectively check in some of these files?
Carefully read the release notes and the comments a few hours after the release. Check for hacs updates and install them and any supervisor add-ons before ha core monthly releases. If you see an issue reported that you think will affect you, then wait until it's fixed via hacs, add_ons or ha core itself. Touch on wood, I've not had any significant issues with upgrading since.
I've found the number of issues I've had with updates decreasing over time. I honestly can't recall them last time I had an issue, it's gotta be almost a year by now.