OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#13838 — p19-90-n6 && p19-91-n6
Scheduled Maintenance Report for Network & Infrastructure
Completed
Around midnight tonight (01/07/15), we will update this 6004 of the web hosting core for 2 reasons: to fix bugs and to support the FEX2348TQ (48x 1g/10g baseT)

The update will be performed in ISSU - Hitless
confirmed by an sh install all impact

# sh install all impact kickstart bootflash:n6000-uk9-kickstart.7.1.1.N1.1.bin system n6000-uk9.7.1.1.N1.1.bin

Verifying image bootflash:/n6000-uk9-kickstart.7.1.1.N1.1.bin for boot variable \"kickstart\".
[####################] 100% -- SUCCESS

Verifying image bootflash:/n6000-uk9.7.1.1.N1.1.bin for boot variable \"system\".
[####################] 100% -- SUCCESS

Verifying image type.
[####################] 100% -- SUCCESS

Extracting \"system\" version from image bootflash:/n6000-uk9.7.1.1.N1.1.bin.
[####################] 100% -- SUCCESS

Extracting \"kickstart\" version from image bootflash:/n6000-uk9-kickstart.7.1.1.N1.1.bin.
[####################] 100% -- SUCCESS

Extracting \"bios\" version from image bootflash:/n6000-uk9.7.1.1.N1.1.bin.
[####################] 100% -- SUCCESS

Extracting \"fexth\" version from image bootflash:/n6000-uk9.7.1.1.N1.1.bin.
[####################] 100% -- SUCCESS

Performing module support checks.
[####################] 100% -- SUCCESS

Notifying services about system upgrade.
[####################] 100% -- SUCCESS



Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
0 yes non-disruptive reset
1 yes non-disruptive rolling
2 yes non-disruptive rolling
100 yes non-disruptive rolling
101 yes non-disruptive rolling
102 yes non-disruptive rolling
103 yes non-disruptive rolling
104 yes non-disruptive rolling
105 yes non-disruptive rolling
106 yes non-disruptive rolling
107 yes non-disruptive rolling
108 yes non-disruptive rolling
109 yes non-disruptive rolling
110 yes non-disruptive rolling
111 yes non-disruptive rolling
112 yes non-disruptive rolling
113 yes non-disruptive rolling
114 yes non-disruptive rolling
115 yes non-disruptive rolling
116 yes non-disruptive rolling
117 yes non-disruptive rolling
118 yes non-disruptive rolling
119 yes non-disruptive rolling
120 yes non-disruptive rolling
121 yes non-disruptive rolling
122 yes non-disruptive rolling
123 yes non-disruptive rolling
124 yes non-disruptive rolling
125 yes non-disruptive rolling
126 yes non-disruptive rolling
127 yes non-disruptive rolling
128 yes non-disruptive rolling
129 yes non-disruptive rolling
130 yes non-disruptive rolling
131 yes non-disruptive rolling
132 yes non-disruptive rolling
133 yes non-disruptive rolling
134 yes non-disruptive rolling
135 yes non-disruptive rolling
136 yes non-disruptive rolling
137 yes non-disruptive rolling
138 yes non-disruptive rolling
139 yes non-disruptive rolling



Images will be upgraded according to following table:
Module Image Running-Version New-Version Upg-Required
------ ---------------- ---------------------- ---------------------- ------------
0 system 7.0(4)N1(1) 7.1(1)N1(1) yes
0 kickstart 7.0(4)N1(1) 7.1(1)N1(1) yes
0 bios v2.6.0(11/21/2012) v2.9.0(12/10/2014) no
0 power-seq v5.0 v5.0 no
0 fabric-power-seq v3.0 v3.0 no
1 power-seq v2.0 v2.0 no
2 power-seq v2.0 v2.0 no
100 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
101 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
102 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
103 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
104 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
105 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
106 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
107 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
108 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
109 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
110 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
111 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
112 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
113 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
114 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
115 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
116 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
117 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
118 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
119 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
120 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
121 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
122 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
123 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
124 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
125 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
126 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
127 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
128 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
129 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
130 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
131 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
132 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
133 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
134 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
135 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
136 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
137 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
138 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
139 fexth 7.0(4)N1(1) 7.1(1)N1(1) yes
0 microcontroller v1.1.0.3 v1.1.0.4 yes

Update(s):

Date: 2015-07-02 10:10:11 UTC
After the ISSU of p19-91-n6, we lost the peer-keepalive between p19-90 and 91.
The peer-link went down at the same time, and there are no logs!

Summary of the update:
p19-91-n6 became isolated => the VPC was behaving normally and didn't cause any impact.
However, a mismatch speed - a bug that should have been fixed with this release - occurred as a result of the peer-keepalive and peerlink remounting.

240serveurs / 2000 have been impacted, which caused interruption to the service on the web hosting infrastructure http://status.ovh.net/?do=details&id=10005

Service is now back up.

We will speak with Cisco tomorrow to continue to post-mortem.

Date: 2015-07-02 10:05:52 UTC
After updating the micro-controller, we need to hard reboot the switches.
Starting with p19-91

p19-91-n6# reload power-cycle

Date: 2015-07-02 10:05:03 UTC
Install has been successful.



e e d8
d8b d8b 888 888 _d88__ 888 888
d888bdY88b 888 888 888 888 888
/ Y88Y Y888b 888 888 888 888 888
/ YY Y888b 888 888 888 888 888
/ Y888b \"88_-888 \"88_/ \"88_-888

NEXUS 6004 MUTU
OVH.com

p19-91-n6 login:

Date: 2015-07-02 10:04:54 UTC
2015 Jul 2 01:33:18 p19-91-n6 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 136 is online
2015 Jul 2 01:33:18 p19-91-n6 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 137 is online
2015 Jul 2 01:33:18 p19-91-n6 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 138 is online
2015 Jul 2 01:33:18 p19-91-n6 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 119 is online
2015 Jul 2 01:33:18 p19-91-n6 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 139 is online
2015 Jul 2 01:33:18 p19-91-n6 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online
[####################] 100% -- SUCCESS

Supervisor non-disruptive upgrade successful.

Date: 2015-07-02 10:04:45 UTC
Module 138: Non-disruptive upgrading.
SUCCESS

Module 139: Non-disruptive upgrading.
SUCCESS

Install has been successful.

Moving on to 91-n6

Date: 2015-07-02 10:04:34 UTC
Module 127: Non-disruptive upgrading.
SUCCESS

Module 128: Non-disruptive upgrading.
SUCCESS

Date: 2015-07-02 10:04:28 UTC
Module 117: Non-disruptive upgrading.
SUCCESS

Module 118: Non-disruptive upgrading.
SUCCESS

Module 119: Non-disruptive upgrading.
SUCCESS

Date: 2015-07-02 10:04:20 UTC
Module 100: Non-disruptive upgrading.
SUCCESS

Module 101: Non-disruptive upgrading.
SUCCESS

Module 102: Non-disruptive upgrading.
SUCCESS

Module 103: Non-disruptive upgrading.
SUCCESS

Module 104: Non-disruptive upgrading.

Date: 2015-07-02 10:04:14 UTC
2015 Jul 2 00:24:36 p19-90-n6 %$ VDC-1 %$ %SATCTRL-FEX127-2-SATCTRL_IMAGE: FEX127 Image update complete. Install pending
2015 Jul 2 00:24:37 p19-90-n6 %$ VDC-1 %$ %SATCTRL-FEX134-2-SATCTRL_IMAGE: FEX134 Image update complete. Install pending
2015 Jul 2 00:24:37 p19-90-n6 %$ VDC-1 %$ %SATCTRL-FEX126-2-SATCTRL_IMAGE: FEX126 Image update complete. Install pending
2015 Jul 2 00:24:43 p19-90-n6 %$ VDC-1 %$ %SATCTRL-FEX132-2-SATCTRL_IMAGE: FEX132 Image update complete. Install pending
2015 Jul 2 00:24:52 p19-90-n6 %$ VDC-1 %$ %SATCTRL-FEX133-2-SATCTRL_IMAGE: FEX133 Image update complete. Install pending
2015 Jul 2 00:24:53 p19-90-n6 %$ VDC-1 %$ %SATCTRL-FEX129-2-SATCTRL_IMAGE: FEX129 Image update complete. Install pending
2015 Jul 2 00:24:55 p19-90-n6 %$ VDC-1 %$ %SATCTRL-FEX138-2-SATCTRL_IMAGE: FEX138 Image update complete. Install pending
2015 Jul 2 00:24:58 p19-90-n6 %$ VDC-1 %$ %SATCTRL-FEX136-2-SATCTRL_IMAGE: FEX136 Image update complete. Install pending
2015 Jul 2 00:25:02 p19-90-n6 %$ VDC-1 %$ %SATCTRL-FEX130-2-SATCTRL_IMAGE: FEX130 Image update complete. [########### ] 50%
2015 Jul 2 00:25:05 p19-90-n6 %$ VDC-1 %$ %SATCTRL-FEX135-2-SATCTRL_IMAGE: FEX135 Image update complete. Install pending
2015 Jul 2 00:25:28 p19-90-n6 %$ VDC-1 %$ %SATCTRL-FEX139-2-SATCTRL_IMAGE: FEX139 Image update complete. [####################] 100%
-- SUCCESS

Module 100: Non-disruptive upgrading.
[# ] 0%

Date: 2015-07-02 10:04:04 UTC
Module 0: Refreshing compact flash and upgrading bios/loader/bootrom/power-seq.
Warning: please do not remove or power off the module at this time.
Note: Power-seq upgrade needs a power-cycle to take into effect.
On success of power-seq upgrade, SWITCH OFF THE POWER to the system and then, power it up.
Note: Micro-controller upgrade needs a power-cycle to take into effect.
On success of micro-controller upgrade, SWITCH OFF THE POWER to the system and then, power it up.
[####################] 100% -- SUCCESS

Upgrade can no longer be aborted, any failure will result in a disruptive upgrade.

Date: 2015-07-02 10:03:57 UTC
Let's rock n roll

0 microcontroller v1.1.0.3 v1.1.0.4 yes


Do you want to continue with the installation (y/n)? [n] y

Install is in progress, please wait.

Date: 2015-07-02 10:03:46 UTC
Starting the upgrade.
Posted Jul 01, 2015 - 14:28 UTC