I always love the Aruba Atmoshpere technical keynotes. Sure, the main keynote on the first day is fun, entertaining, and well-orchestrated. Like a professional keynote should be. This year, one of my favorite presenters, Guy Kawasaki was part of the lead Keynote session.
But my favorite has always been the technical keynotes on the second day. This is where Aruba gets to show off it’s latest and greatest tech and do live demos of up and coming features. This year was no exception. (OK I did miss having Keerti doing the technical presentations – but this year with his promotion to General Manager/SVP – he moved to Emcee day one and Partha Narasimham as CTO did those duties on the technical keynote).
There were demos, and views of various new technologies Aruba has in its portfolio and those that are soon to be products. That was great. But the thing that knocked my socks off was the gutsy move to do a live upgrade on the conference Wireless LAN – including rebooting of controllers and AP’s pushing new code DURING the technical keynote while thousands of folks were actively working on the network!
I’m not sure if most those in attendance were as suitably impressed with this feat. Most didn’t even notice any issues – because the process proceeded smoothly. Of course, it didn’t go completely error-free… but for the most part it was pulled off without a hitch. OK… lets’ talk about the hitch…
As a setup to this situation, on the previous day, during the main keynote while I was doing some live blogging, I found two of my devices worked perfectly, but one could just not pull a DHCP address. I thought that odd… no matter what I did, one device refused to get anything but an APIPA address (169.254.X.X). Meaning it was connected and associated to the Wi-Fi, but not able to get a response from the DHCP server. Confused… but not overly alarmed, I just continued to work on the other devices.
Now fast-forward to the Technical Keynote, and the massive and disruptive process of upgrading firmware on controllers and access points during live network event. My devices worked positively and completely through this without a hitch, yet someone less than a meter away was having lots of connection issues… LOTS of connection issues. No matter what they did, their connections had a difficult time staying associated to the same access point.
To Aruba’s credit – directly after the session we had a nice private interaction with Jone Ostebo, Aruba’s Solution Engineering Manager, who explained both issues to us. After understanding what the issues were, I was even more impressed with how well they professionally handled the issues. First, they were running beta code in a production environment with all those inherent chances for failure. Second, doing some gutsy moves upgrading on a live production network with thousands of users all watching for any possible flaws. Well done!
For the first issue with the DHCP, we learned there was a ‘fat finger’ incident which left a couple characters in a line of code transposed, which caused the DHCP issue whereby specific MAC addresses were not being responded to, and all others were. Within 24-hours they had new code, tested it on a sample network, then install the fix on the production network and we had none of those issues on day two.
As for the second issue – prior to the upgraded firmware push, the clients in the auditorium were load balanced across the access points –during the software upgrade of the access points some clients were not load balanced correctly. The load balancing code in the beta software wasn’t tuned yet, and to many clients got ‘stuck’-with over 170 on a single radio, and thus the connection issues.
Again – gutsy move to do a live upgrade – but also totally understandable after finding the cause. Within minutes of the issue, they’d isolated it, and understood the problem. All while a live technical keynote with all its demonstrations were going on at the same time. Kudos to the technical team at Atmosphere 2017. I was duly impressed! You are rock stars in my book.
Take away: if Aruba is willing to test their beta code in such a fashion and pull it off… I think this new feature of live upgrades will be worthy of end-customers to start using as soon as this updated product ships.
