OVHcloud Network Status
Current status
Legend
- Operational
- Degraded performance
- Partial Outage
- Major Outage
- Under maintenance
FS#12244 — bhs2-11a-n6
Incident
Report for Network & Infrastructure
Resolved
This nexus has rebooted.
Reason: Reset triggered due to HA policy of Reset
Packs are forwarded by bhs2-11b-n6.
Update(s):
Date: 2014-12-18 01:35:42 UTC
The pair is stable since 30 minutes.
Date: 2014-12-18 01:35:24 UTC
The only possible workaround was to reload the 4 FEXS in question.
We don't see ports err-disable anymore.
Date: 2014-12-18 01:34:13 UTC
All fex are up to date.However we still have err-disable port. We are checking the issue with cisco.
Date: 2014-12-18 01:31:01 UTC
The fex is still not recognized. We rebooted bhs2-11b-n6.
Date: 2014-12-18 01:30:36 UTC
install failed.
We replaced fex 101.
Date: 2014-12-18 01:30:16 UTC
We restarted the upgrade:
Images will be upgraded according to following table:
Module Image Running-Version New-Version Upg-Required
------ ---------------- ---------------------- ---------------------- ------------
1 system 6.0(2)N2(5) 6.0(2)N2(5) no
1 kickstart 6.0(2)N2(5) 6.0(2)N2(5) no
1 bios v1.5.0(12/29/2012) v1.5.0(12/29/2012) no
1 power-seq v4.0 v4.0 no
1 fabric-power-seq v4.0 v4.0 no
2 power-seq v4.0 v4.0 no
100 fexth 6.0(2)N2(5) 6.0(2)N2(5) no
101 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
102 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
103 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
104 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
105 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
106 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
107 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
108 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
109 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
110 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
111 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
112 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
113 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
114 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
115 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
116 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
117 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
118 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
119 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
120 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
1 microcontroller v1.2.0.5 v1.2.0.5 no
Additional info for this installation:
--------------------------------------
Remove QoS & ACL config on L3 interfaces and SVIs if any
Service \"vpc\" : STP Preupgrade Check failed on VPC peer switch
Service \"lacp\" : LACP: Upgrade will be disruptive as timeout on fex lacp_issu req
Do you want to continue with the installation (y/n)? [n] y
Install is in progress, please wait.
Date: 2014-12-18 01:29:32 UTC
The fex101 failed during the hot swap update :
Module 101: Non-disruptive upgrading.
FAIL. Return code -1.
Remaining action::
\"Module(s) 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120 still need to be upgraded\".
Install has failed. Return code 0x40930020 (Non-disruptive upgrade of a module failed).
Please identify the cause of the failure, and try 'install all' again.
Date: 2014-12-18 01:28:51 UTC
We always have ports with err-disable. We have updated the n6 pair in hot swap.
Date: 2014-12-18 01:25:42 UTC
Several ports on the fex113 (no luck...) passed to err-disable and we could not shut/no shut them,commands fit but they are not applied.
bhs2-11a-n6(config-if)# shut
Please check if command was successful using appropriate show commands
The only way to reset them : reload for the switches one by one.
There is no impact on servers that are already up (it will switch on the second switch of the pair).
Date: 2014-12-17 18:18:00 UTC
We will update them on 6.2.N2.5.
Date: 2014-12-17 18:17:26 UTC
bhs2-11b-n6 has rebooted too.
Reason: Reset triggered due to HA policy of Reset
Packs are forwarded by bhs2-11b-n6.
Update(s):
Date: 2014-12-18 01:35:42 UTC
The pair is stable since 30 minutes.
Date: 2014-12-18 01:35:24 UTC
The only possible workaround was to reload the 4 FEXS in question.
We don't see ports err-disable anymore.
Date: 2014-12-18 01:34:13 UTC
All fex are up to date.However we still have err-disable port. We are checking the issue with cisco.
Date: 2014-12-18 01:31:01 UTC
The fex is still not recognized. We rebooted bhs2-11b-n6.
Date: 2014-12-18 01:30:36 UTC
install failed.
We replaced fex 101.
Date: 2014-12-18 01:30:16 UTC
We restarted the upgrade:
Images will be upgraded according to following table:
Module Image Running-Version New-Version Upg-Required
------ ---------------- ---------------------- ---------------------- ------------
1 system 6.0(2)N2(5) 6.0(2)N2(5) no
1 kickstart 6.0(2)N2(5) 6.0(2)N2(5) no
1 bios v1.5.0(12/29/2012) v1.5.0(12/29/2012) no
1 power-seq v4.0 v4.0 no
1 fabric-power-seq v4.0 v4.0 no
2 power-seq v4.0 v4.0 no
100 fexth 6.0(2)N2(5) 6.0(2)N2(5) no
101 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
102 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
103 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
104 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
105 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
106 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
107 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
108 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
109 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
110 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
111 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
112 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
113 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
114 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
115 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
116 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
117 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
118 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
119 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
120 fexth 6.0(2)N2(3) 6.0(2)N2(5) yes
1 microcontroller v1.2.0.5 v1.2.0.5 no
Additional info for this installation:
--------------------------------------
Remove QoS & ACL config on L3 interfaces and SVIs if any
Service \"vpc\" : STP Preupgrade Check failed on VPC peer switch
Service \"lacp\" : LACP: Upgrade will be disruptive as timeout on fex lacp_issu req
Do you want to continue with the installation (y/n)? [n] y
Install is in progress, please wait.
Date: 2014-12-18 01:29:32 UTC
The fex101 failed during the hot swap update :
Module 101: Non-disruptive upgrading.
FAIL. Return code -1.
Remaining action::
\"Module(s) 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120 still need to be upgraded\".
Install has failed. Return code 0x40930020 (Non-disruptive upgrade of a module failed).
Please identify the cause of the failure, and try 'install all' again.
Date: 2014-12-18 01:28:51 UTC
We always have ports with err-disable. We have updated the n6 pair in hot swap.
Date: 2014-12-18 01:25:42 UTC
Several ports on the fex113 (no luck...) passed to err-disable and we could not shut/no shut them,commands fit but they are not applied.
bhs2-11a-n6(config-if)# shut
Please check if command was successful using appropriate show commands
The only way to reset them : reload for the switches one by one.
There is no impact on servers that are already up (it will switch on the second switch of the pair).
Date: 2014-12-17 18:18:00 UTC
We will update them on 6.2.N2.5.
Date: 2014-12-17 18:17:26 UTC
bhs2-11b-n6 has rebooted too.