A dedicated model that’s fine tuned for specific tasks and then the temperature set to not allow variation is the key. If you want some variation, have a separate agent/model do that part, but some things you don’t want creativity once you’d defined what the job is and have it working. I think the weak point in all that you mentioned is still the voice detection. It doesn’t take much to break understanding of a clear command, much less some not said well or low or noise in the room. At best you have the system say it didn’t get that and to try again. Don’t let it try and figure out what you said, that will end badly at some point.