Some really strange stuff here while updating a PE R720! I tried updating the PSU along with the other hardware during within the Firmware Update in the LCC. It seemed to not have any issues with all the other firmware updates until it hit the PSUs. Then, oddly enough, when I attempted to update the PSUs my phone would start ringing with my users not being able to connect to any of my other servers. !!! I couldn't find the problem and it started working again in about a minute or two. I just considered it some kind of "network hiccup" and left it at that. BUT I noticed that the PSUs failed fully update, so I attempted the update again ... BAM! Network down, users calling, and PSUs failed to completely update to the newest version again... Coincidence ... maybe...
SO, I had another server to do. I did all the other updates, waited until a slow period during the day, started some pings, and then updated the PSUs ... BAM all pings failing and network is down. I ran to the server room to see if I could determine if the server Gbe switches (2x PC6248) crashed but it wasn't obviously rebooting. I then dug into the switch logs and it looks like it forces my second switch to become the the STP root and proceeds to freak out:
| Spanning Tree Topology Change: 0, Unit: 1 |
| 2/0/22 is transitioned from the Learning state to the Forwarding state in instance 0 |
| Unit 2 elected as the new STP root |
| Instance 0 has elected a new STP root: 8000:d067:e583:55c0 |
| Spanning Tree Topology Change: 0, Unit: 1 |
| Instance 0 has elected a new STP root: 8000:a4ba:db56:b4bb |
| Spanning Tree Topology Change: 0, Unit: 1 |
| Spanning Tree Topology Change: 0, Unit: 1 |
| Unit 2 elected as the new STP root |
| Instance 0 has elected a new STP root: 8000:d067:e583:55c0 |
| Instance 0 has elected a new STP root: 8000:a4ba:db56:b4bb |
| Unit 2 elected as the new STP root |
| Instance 0 has elected a new STP root: 8000:d067:e583:55c0 |
| Instance 0 has elected a new STP root: 8000:a4ba:db56:b4bb |
| Unit 2 elected as the new STP root |
| Instance 0 has elected a new STP root: 8000:d067:e583:55c0 |
| Instance 0 has elected a new STP root: 8000:a4ba:db56:b4bb |
| Unit 2 elected as the new STP root |
| Instance 0 has elected a new STP root: 8000:d067:e583:55c0 |
| Instance 0 has elected a new STP root: 8000:a4ba:db56:b4bb |
| Link Down: 1/0/12 |
| Link on 1/0/12 is failed |
| Link Down: 1/0/44 |
| Link on 1/0/44 is failed |
| 1/0/12 is transitioned from the Forwarding state to the Blocking state in instance 0 |
| 1/0/44 is transitioned from the Forwarding state to the Blocking state in instance 0 |
| Link Down: 2/0/10 |
| Link on 2/0/10 is failed |
| 2/0/10 is transitioned from the Forwarding state to the Blocking state in instance 0 |
| Link Down: 2/0/44 |
| Link on 2/0/44 is failed |
| Link Down: 2/0/22 |
| Link on 2/0/22 is failed |
| 2/0/44 is transitioned from the Forwarding state to the Blocking state in instance 0 |
| 2/0/22 is transitioned from the Forwarding state to the Blocking state in instance 0 |
| Link Up: 1/0/12 |
| Link Up: 1/0/44 |
| 1/0/12 is transitioned from the Forwarding state to the Blocking state in instance 0 |
| 1/0/44 is transitioned from the Forwarding state to the Blocking state in instance 0 |
| Link Up: 2/0/10 |
| 2/0/10 is transitioned from the Forwarding state to the Blocking state in instance 0 |
| Link Up: 2/0/22 |
| 2/0/22 is transitioned from the Forwarding state to the Blocking state in instance 0 |
| Link Up: 2/0/44 |
| 2/0/44 is transitioned from the Forwarding state to the Blocking state in instance 0 |
| 1/0/12 is transitioned from the Learning state to the Forwarding state in instance 0 |
| 1/0/44 is transitioned from the Learning state to the Forwarding state in instance 0 |
| 2/0/10 is transitioned from the Learning state to the Forwarding state in instance 0 |
| Spanning Tree Topology Change: 0, Unit: 1 |
| 2/0/22 is transitioned from the Learning state to the Forwarding state in instance 0 |
| 2/0/44 is transitioned from the Learning state to the Forwarding state in instance 0 |
| Spanning Tree Topology Change: 0, Unit: 1 |
| Spanning Tree Topology Change: 0, Unit: 1 |
| Spanning Tree Topology Change: 0, Unit: 1 |
| Spanning Tree Topology Change: 0, Unit: 1 |
| Spanning Tree Topology Change: 0, Unit: 1 |
| Spanning Tree Topology Change: 0, Unit: 1 |
| Spanning Tree Topology Change: 0, Unit: 1 |
| Spanning Tree Topology Change: 0, Unit: 1 |
Then and all is well again...
Whaaaaa? I have no idea why updating the PSU firmware on the server would cause this. I can remove power from the server completely and it doesn't cause this problem. Could it cause a surge on the PDU that the switch and Server is plugged into?
Also strange is that the PSUs didn't update to the latest version available from the Dell FTP server. LCC log shows Version change from previous version 69.45.99 -> Current version 69.45.9B ... so does that mean I have to go in steps instead directly to 6B.47.9B ... HOWEVER that I can somewhat understand ... the network madness I cannot.
Any ideas? Is there some ghost in the PSU firmware? (It is close to Halloween) OOohhh Spooky ... yea actually when it takes out my network ... I'm pretty spooked!!!!!