Behind-the-scenes of the Windows 10 Creators Update Bug Bash
Hi! My name is Anita George, and I am the leader of the two Bug Bashes for the Windows 10 Creators Update (the one in November 2016 and the one we just finished up in Feb). For this most recent Bug Bash, Windows Insiders completed approximately 108,900 Quests and submitted or upvoted an astounding 115,100 feedback items! Thank you so much to everyone that took the time to Bug Bash with us. The level of participation was just incredible! So what goes into running a Bug Bash? I’d like to take some time to explain the behind-the-scenes work that happens to make a Bug Bash successful.
I can’t take full credit for running the Bug Bashes. There is a team of people from across the Windows and Devices Group (WDG) that help with prepping and running an event of this magnitude. We call them the “Bug Bash Crew”. These people are responsible for external Bug Bash prep like driving Quest creation, setting up webcasts, monitoring data, designing the Bug Bash badge, and sending out surveys. We have representatives from some of the key areas in WDG like Shell, Apps, Surface, Xbox, IoT, HoloLens, Windows Insider Program, Localization, and Feedback Hub. Below is a picture of our Bug Bash Crew with some familiar names for those of you that participated in one of our Beam webcasts.
Bug Bash Crew + twitter handles: (left to right) Anthony Kinsey (@AKinseyMSFT), Aman Benes (@Aman_msft), Jen Gentleman (@JenMsft), Kyle Hagel, Aiko Diamond, Josh Moore, Sunitha Gowda, Anita George (@anitag_MS), Jason Howard (@NorthFaceHiker), Srini Malladi (@srinivm_MSFT), and Reid Szafraniec (@ReidMSFT).
We also do a lot of work to help our internal WDG employees to get involved in Bug Bashes and this participation spans across 27 buildings on Microsoft campus and one building in Vancouver, BC. We provide an internal wiki (website) with key information about the Bug Bash and show regular daily updates on how the Bug Bash is progressing through a reporting dashboard.
Internal employees across the different teams in WDG work to create and review Quests while members of the Bug Bash Crew are responsible for publishing these Quests at the right time and for the correct builds and devices. The Quests do not get authored by the BugBash Crew. When you are running through a Quest, you are performing steps that an engineer from that specific feature team has written down in the hopes that someone will exercise his/her feature. We have some Quests that we want to make available on certain days, like after the second Bug Bash build goes out. We group Quests based on some themes like “apps”, “devices”, and “quality” so that participants can focus on similar functionality for that day. Many feature teams want their Quests to show up on the first day of the Bug Bash, but we found doing this in small, themed groupings is more consumable than getting 232 Quests all on one day, for example.
Our Microsoft employee participants also need to know where internally to install the latest builds which can be complex since we have engineers running in different rings (like Canary and Selfhost) and some will run local builds of Windows with preliminary code that hasn’t made it into a flighted build. Over the course of the last two Bug Bashes, we have shifted our focus to the Feedback Hub and now all Microsoft employees participating in the Bug Bash internally enter their issues into Feedback Hub just like Windows Insiders. During the Bug Bash and after it ends, we triage the items in Feedback Hub as well as the bugs that get moved into Visual Studio. We try to complete this triaging within 2-3 weeks after the end of the Bug Bash.
Every Bug Bash, we host an internal Microsoft event to generate new and interesting bugs. For the Bug Bash in November, we set up a Device Room which contained about 10 Surface Studios, a bunch of IoT devices, and new performance Surface Books. Employees could stop by during the 4-hour event to try running Quests on this unique hardware. For the February Bug Bash, we tried something much different inspired by all the work we’ve done in emerging economies in Africa and other parts of the world. We set up a limited connectivity network in one of our campus buildings to simulate what many people in the world experience when trying to use technology over the internet. We called it Nigeria Day, hosted by Dona Sarkar, and we had good African food and music as people tried running Quests with low connectivity.
Participants in Nigeria Day: (left to right) Tommy Lasebikan, Fernando Sanchez Gonzalez, Jeremiah Marble, Bambo Sofola, Wale Martins, Dona Sarkar, Shina Oyetosho, Ayuba Audu.
One key area we focus on during the Bug Bash is collecting data to determine if the Bug Bash is progressing as expected. In the past, this was time-consuming as we pulled data from multiple places with multiple data-science experts looking at it and validating it. Between the November and February Bug Bashes, we had a small team of engineers work on creating a Power BI dashboard that would show progress on Quests completed, feedback items entered, bugs triaged, and whether teams and individuals inside of WDG were participating more than others. We did this to form a healthy competition between the 78 Windows Engineering Directors and their teams. Overall, this was a huge time savings for the Bug Bash Crew and was used heavily by the engineering teams. We also monitor the progress of our Windows Insider activity during the Bug Bash with this dashboard. Do you know that the most completed Quest was the one for our new Night light feature? And the Apps and Games category received the most feedback items from Insiders? Outside of the United States, Germany was the most active country to provide feedback, followed closely by India in second place. Way to go, Germany and India! We are looking to see what parts of this dashboard we can provide externally to Insiders for the Bug Bashes in the future. Wouldn’t that be fun?
If you have feedback on the February 2017 Windows 10 Bug Bash, let us know through Feedback Hub. I hope this information gives you some insight into what we do inside of WDG Engineering to provide you with an optimal Bug Bash experience.
Thank you everyone for participating. We couldn’t have done it without you!
Anita George.
#winbugbash #neverstopquesting
Source: Behind-the-scenes of the Windows 10 Creators Update Bug Bash
Leave a Reply