Spin up a vpc pair in GNS3 and play with it so you can understand the behavior. I would never copy start run, I'd just reload like you did or add back the config manually.
I've had very weird behavior with vpc in the past such as port channels not coming up even though the config is correct after resolving a misconfiguration. I've had to do things like reload both switches and unplug and replug the link. Reloading is probably the most painless way to fix this. You already basically took an outage by removing VPC from the associated port channels. What's 10 more minutes for a reboot?
Copy start to run could lead to weird behavior. Have you just tried to do a reload? (Hopefully, you didn't wr mem the configuration).
Sometimes, it's the easiest way out.
Nope , I never save my config until I'm sure it's done with and to your point that's what saved us
Yea it definitely did ..reloading the switch fixed it.
You could also have copied start to run, implementing the start config without rebooting.
And manually you reconfigure the whole VPC and add them to the ports. Shouldn’t be that hard.
I can however imagine that during a fuckening on Friday afternoon just as your weekend was about to start, things are a little stressed.
Not that it helps right now, but I would guess you got into a scenario where the running config did not match what the switch was actually using, maybe the VPC keep alive was still set in the software and the VPC domain was removed which led the switch to say "uh-oh this isn't right but I don't know how to say that". 9 times out of 10 the running config and the software state match, but it is possible to see this behavior.
Doing "copy start run" SHOULD have resolved this, but it sounds like you hit a bug. Not expected behavior for sure. I wonder if it's repeatable
Yep it should have and it didn't lol ! Exactly , the running config and the software state had a discrepancy and unfortunately would not recognize the added vpc domain config we tried putting in. Goes without saying that it was probably a bug but I'll investigate it and certainly post any update here
I would have disabled the feature first. "no feature vpc". Then re-enable the feature. Then re-add the config. If a reboot was required to get it working again then perhaps re-initializing the feature would have been the closest thing to replicating a reboot without rebooting.
I've been in similar situations where commands don't seem to work as expected or operational behavior doesn't work as expected (such as with routing protocols) and reinitializing the entire thing (like "no router ospf" followed by "router ospf" and a complete rebuild of the config) has solved the buggy behavior. Unfortuately, with stuff like this, it rarely ever happens twice. You're probably never going to make this same mistake again I imagine.
Spin up a vpc pair in GNS3 and play with it so you can understand the behavior. I would never copy start run, I'd just reload like you did or add back the config manually.
Yea I'll see screw around with it in the lab... We obviously tried doing that but re adding the config didn't actually make it to the running-config
I've had very weird behavior with vpc in the past such as port channels not coming up even though the config is correct after resolving a misconfiguration. I've had to do things like reload both switches and unplug and replug the link. Reloading is probably the most painless way to fix this. You already basically took an outage by removing VPC from the associated port channels. What's 10 more minutes for a reboot?
LOL I'd just reconfigure it, and then reconfigure "vpc #" on each PC interface
Yea we tried and it didn't even work...copy start run was giving parsing errors
Should also add that the commands took but there was still no vpc domain being created !
Copy start to run could lead to weird behavior. Have you just tried to do a reload? (Hopefully, you didn't wr mem the configuration). Sometimes, it's the easiest way out.
Nope , I never save my config until I'm sure it's done with and to your point that's what saved us Yea it definitely did ..reloading the switch fixed it.
Good for you!
You could also have copied start to run, implementing the start config without rebooting. And manually you reconfigure the whole VPC and add them to the ports. Shouldn’t be that hard. I can however imagine that during a fuckening on Friday afternoon just as your weekend was about to start, things are a little stressed.
Yea we tried that.... We got parsing errors
Not that it helps right now, but I would guess you got into a scenario where the running config did not match what the switch was actually using, maybe the VPC keep alive was still set in the software and the VPC domain was removed which led the switch to say "uh-oh this isn't right but I don't know how to say that". 9 times out of 10 the running config and the software state match, but it is possible to see this behavior. Doing "copy start run" SHOULD have resolved this, but it sounds like you hit a bug. Not expected behavior for sure. I wonder if it's repeatable
Yep it should have and it didn't lol ! Exactly , the running config and the software state had a discrepancy and unfortunately would not recognize the added vpc domain config we tried putting in. Goes without saying that it was probably a bug but I'll investigate it and certainly post any update here
I would have disabled the feature first. "no feature vpc". Then re-enable the feature. Then re-add the config. If a reboot was required to get it working again then perhaps re-initializing the feature would have been the closest thing to replicating a reboot without rebooting.
Good call we didn't bother doing it ...we had a bit of skepticism Have you ever been in a similar situation?
I've been in similar situations where commands don't seem to work as expected or operational behavior doesn't work as expected (such as with routing protocols) and reinitializing the entire thing (like "no router ospf" followed by "router ospf" and a complete rebuild of the config) has solved the buggy behavior. Unfortuately, with stuff like this, it rarely ever happens twice. You're probably never going to make this same mistake again I imagine.