WWDC: Putting polish on the Apple

I was lucky enough to go to Apple’s Worldwide Developer’s Conference (WWDC) in San Francisco a few weeks ago.  It was a great event, and I hope to make the trip to San Francisco again next year.  I’ve been to a few other technical conferences, and some nice touches at WWDC that I’d like to see everywhere are:

• Power plugs in the presentation rooms.  Seriously, people, this is the 21st century, and everyone has a laptop.
• Feed me.  Constantly.  A ready supply of juice and coffee will keep your conference attendees from getting crabby.
• Record the sessions and the Q-A at the end of each presentation.  Apple will be making all of this available to attendees in a few weeks, which is great because I had to miss a few sessions.
• Open lab sessions with Apple developers, not just marketing mouthpieces.  It’s a slick move and a show of confidence in your people to let attendees drop by with their source code and start asking questions.

Because of NDA stuff I can’t really divulge any details of the sessions, but I’m really excited to be working with OS X, and Apple did a lot to emphasize that we’re part of a development community.  I think Apple’s approach of leveraging open standards (and open source) and controlling hardware has given them a real advantage with quality and user experience–I can’t wait to see what they come up with next!

Wisdom passed on

From some of my favorite blogs, I’ve run across these interesting posts on testing and feedback:

Joel on Software: Top Twelve Tips for Running a Beta Test

Joel has some great points, especially about how long to run a beta for (#3 and #6), and how to consistently get feedback throughout the beta process (#11). During the betas that I ran, I noticed, just like Joel, that lots of people didn’t provide feedback. His solution to this problems is to get LOTS of people to evaluate the software (sort of a shotgun approach), but to me, that makes it difficult to manage. Especially at smaller software firms, there usually isn’t a person dedicated to analyzing all of the feedback from a beta, so even 100 very active users would be overwhelming. I made Prefinery because I think that if you improve the feedback channel and have the opportunity to hand-pick who can participate in the testing, you’ll get the same quality of feedback with much, much less effort.

shanecrawford.org: Breakfast of Champions

I’ve only recently found Shane’s blog, but I think he makes some great points. Not only do you have to be receptive to feedback (and have thick skin!), the person giving the feedback has to be constructive and provide useful information. Surveys are a great way to get that information because you can guide people down the path of giving you additional comments. Something like: “You didn’t like feature X? How would you have it work instead?” If you give people the opportunity to suggest something new and they say nothing, they are much less likely to complain later on. And if you incorporate their idea, that customer will love you forever.

Roles in system testing

System tests are important because as software developers, especially on a larger team, it’s easy to focus exclusively on your component, your piece of the puzzle in the software release, and lose sight of the bigger picture. System tests take a step back and ask important product-level questions, considering how well disparate components work simultaneously, how consistent the interface and APIs are, and how easy it is to use the product–a critical aspect of software design that is too often overlooked.

But who are the best people to design and conduct system testing? Your software is probably specialized for a particular industry, or it at least serves a pretty specific purpose. Presumably your company has expertise in that field, but junior engineers (yes, the ones most often tasked to write tests) probably haven’t gained insight into how the product they work on is realistically used by customers. More experienced developers, marketing, and salespeople may be better able to understand customer use cases, but their time is often dedicated to strategic initiatives, not test development. While it may not be most useful for these experts to implement system tests, it is beneficial for them to contribute to test plans, and at least ensure they make sense.

Additionally, your customers are primarily concerned with using your whole product, and them running through some core use cases will almost certainly have better than expected code coverage, and more importantly, their testing will emphasize often-used features and is likely to take advantage of real-world data sets. If you find customers interested in sampling the new product before release, you can prevent potentially costly headaches if they uncover problems before you go through the process of finalizing your product.

The testing I’m referring to is of course beta testing, when you let key customers try out your software ahead of time. I feel the beta testing feedback loop is so important in software development that I’ve help launch Prefinery, a beta testing management site. It aims to make the feedback loop more efficient (and useful).

This isn’t to say that you should leave all system testing to customers–that would be asking for trouble. What I’m saying is that customer-driven system tests are deeply important, and a key addition to system testing conducted internally. It is possible to release a product without showing it to customers ahead of time, but running a customer beta program can give you more confidence in your release, and less risk of a patch or recall.