Apple at 50 (Part 3): What I did at Apple
Do you want to spend the rest of your life selling sugared water, or do you want a chance to change the world? — Steve jobs pitch to John Sculley
When I joined Apple in 1989, John Sculley was CEO. At employee orientation I was handed a copy of his book Odyssey. I might still have it in a box around here somewhere. At the time I was working at Sun Microsystems doing technical support, and my boss asked me if I wanted to join him. As a well known Apple nerd, I jumped at the opportunity, and so I was part of the creation of Apple’s first paid technical support team.
That was the Direct Response Center, created because it had a product it knew couldn’t be handled with the standard “Talk to your dealer” support model. That product was A/UX, a System V based UNIX System that was wrapped with the Mac interface.
Over time the scope of the DRC expanded to include other products, many of which were part of Apple’s ongoing (and generally ignored) attempts to gain traction in the enterprise markets. Did you know that Apple once shipped a 3720 emulator system for the Mac? Another one as AppleSearch, intended as a small business search engine system — good luck finding ANY online reference to this one, folks (and it submerged again but ultimately became the pieces that became Spotlight later on).
One of the other enterprise tools we supported was Data Access Language, a “middleware” tool intended to connect into enterprise data sets. This ultimately got merged into System 7 as Data Access Manager, which is the only time in my tenure at Apple a project I was involved in ended up part of an Apple software release. Since nobody in the DRC knew databases, I took on the task to learn it, which ultimately turned me into a decent MySQL DBA.
Along the way Apple transitioned to Michael Spindler, who proceeded to try to get the finances under control. Besides laying off a bunch of people he announced he was creating an outpost in Austin and exiling various groups out there. Well, he didn’t phrase it that way, but you can guess how well that played with people seeing their jobs relocated.
Of course, the DRC was one of the groups going to Austin. I, in my quiet and careful way of dealing with sensitive political matters, made it very clear my chair was moving but I wasn’t. I found out later my bosses (not the boss I came from Sun with, as he’d moved into Developer Relations) wrote me up for that one but neglected to tell me they had, which they were supposed to.
In any event, if your tech support team no longer has any access to engineering resources, you need to create a group to support the support team, so a new 2nd tier team was created in Cupertino to support and train the support teams in Austin, and I transitioned to that, so I mostly kept doing the same thing, except I spent less time on the phone.
Around this time a new organization was created known as Apple Server Marketing, under Jim Groff. This group was responsible for the Mac server line, known as the -50’s: machines like the Mac 6160, 8150, and 9150. I had an opportunity to join them (as something between their webmaster and their token geek), the first time I was directly involved in a marketing team, which I loved.
These machines were hated by various groups within Apple, because unlike a lot of Macs, they sold pretty well and had much better margins than consumer Macs did at the time, and because they weren’t under control of the MacOS group, which released software for them rather reluctantly. The 6150 and 8150 were basically repackaged Macs with upgraded components like server grade hard drives, while the 9150 was a unique design. All of them were pretty good machines for the day. We were now into the PowerPC era of Macintoshes.
Around this time, Spindler killed A/UX. He decided, in another attempt to get traction in the Enterprise markets, to start creating strategic alliances, and the alliance with IBM meant that we woke up one morning to find out were were now an AIX shop. A/UX had been ported to PowerPC and ran quite well, but because of this, it had no future.
In 1994 Apple shipped another interesting product, the Macintosh Application Environment. This was, effectively, a version of MacOS 7.1 built to run on Sun SPARC-based computers. This worked pretty well, but indirectly it set some things going that would shape the rest of my time at Apple.
While trying to figure out how to help people running MAE, it was decided to create a mailing list for MAE users to use to help discuss the product and support each other. Since I was known as an email nerd (I was running a server with a bunch of non-business lists on it on the side), I was asked to put this together. Once that list was up and running, of course, we started getting requests for other lists, and so it didn’t take long before my job became both doing the work needed by the Server Marketing team and running the list server. Around this time I also got involved with the Mailman open source project (the list server I was running), and I was told I could spend up to 20% of my work time supporting that project, which I think was the first time Apple formally supported an open source project.
In 1996 the server marketing group finally released one of the most interesting and weird products ever shipped by Apple: the Apple Network Server. It ran AIX, it was intended for data center operations, had redundant power supplies and was built like a tank (and weighed like one). Oh, and data center focus or no, it wasn’t rack mountable, although ultimately a bracket was created for one.
At some level, it was a fascinating machine, at another, it was a classic “design by committee” system. It was originally intended to run, of all things, Windows NT (no, seriously) — until Spindler found out about it. And since Spindler saw Microsoft as the anti-Christ of computing companies, I wish I’d been in the meeting where he found out about that one. It didn’t run MacOS, because the OS team refused to support it (okay, we did have MacOS running on it, but we were never allowed to ship it). So it became an AIX machine because Apple was now an AIX shop.
They were interesting boxes — Apple ran much of their data center on them, and I believe actually used more of those machines in production than they sold to other companies. I kind of liked it, but it wasn’t really a great data center machine and while AIX is perfectly okay, I always wonder what it would have been like running A/UX instead.
This was the time period when I found out I was going to be laid off in one of Apple’s recurring layoffs. In talking to my boss, he said he put me on the list because he thought I’d want to be and because he knew I’d have no problem getting a new job (thanks for the confidence!). I expressed the thought that I wanted to stay because I thought we could still make things work — and he took me off the list again, so I stayed with Apple (with no regrets). An interesting trivia point on this is that after that layoff, Apple changed the layoff package and if I’d been laid off later, I’d have been about $30,000 the poorer in the package, but that fortunately never happened.
It was become clear the mailing list work was taking over my time from my actual job in server marketing, and it was clear the mailing lists really ought to be a corporate function, so discussions started about transferring the list server (and me) into Apple’s IT group, IS&T. For various reasons this took about six months, but ultimately, I moved into IS&T.
And in grand Apple style, I walked in to meet with my new boss, and he sat me down to let me know they’d been talking, and it made sense for me to actually report to someone else — who I had never met or talked to. And so that’s how I started my life in Apple’s IT teams, where I stayed until I existed Apple a decade later.
As it turned out, my new manager and his boss (the director) were both great managers, although IT was a big project group focussed on Oracle, WebObjects and big iron hardware, and I was this quick-build skunkworks open source nerd. They had been trying to build an internal list server around a commercial server package, but despite a couple of consultant, hadn’t figured out how to get it all integrated. I took the project on and basically threw the server out and built one from scratch that would download the data and turn it into lists on the fly. My director and I had multiple discussions about software without support contracts, but in reality, the support contract hadn’t gotten the system functioning, and mine worked — it worked really well, and was still in use years after I left Apple, and when I left, it was supporting 30,000+ separate lists and thousands of emails distributed a day. And my director slowly started getting comfortable with tools without support contracts.
The list server I brought with me became lists.apple.com, which was still up and functioning up to about five years ago. At its peak it had somewhere around 75 lists for various products and technologies (all of this ultimately transitioned into web forums well after I left). Lists was, initially, a 9150, by the way.
Also in 1996, Apple punted Mike Spindler to the curb and replaced him with Gil Amelio. While I have nothing good to say about Spinlder, I think Amelio gets a bit of a bad rap. He wasn’t a great CEO, but he did some good work stabilizing the company, which at that point was pretty clearly in a death spiral. He also put more focus into QA and testing, so the quality of software and products started to improve, and he realized that the attempt to reinvent the OS — Copland — was no where near functional and wasn’t going to be any time soon, and started looking at options. Taking a close look at Gassee’s Be, and ultimately decided (correctly, I believe) it wasn’t the answer, either, and instead turned to, of course, Steve Jobs and NeXT.
And so set up both the return of Steve, and a future where Apple becomes what we have today. And my world was about to be crazy in a fun but stressful way.
Shortly after Steve returned, I got an email from my old boss in the server marketing team, letting me know the entire team had been laid off and they were all exiting the building, and that they’d left me some stuff in my server room. At the time, my list server was in a small server room (really more of a locked closet) in De Anza 3, because we’d not gotten around to moving it into IS&T facilities. I went up to the old server marketing area in DA3 and it was dark and empty. That was the end of the Apple Network Server.
When I went down to the server closet, I found four huge boxes plus some other hardware bits and bobs — and a property pass. In those boxes were four demo units for an unreleased product that were about to go out for evaluation — the second generation ANS servers, code named Deepdish. These would have been good data center machines. It was more hardware than I was comfortable taking ownership, so in talking to my bosses, I gave two of them to IS&T, where they got used internally for a while. I took the others home, and in fact they ran me home server for about five years. It turned out all four units had their power supplies fry after 19-24 months of service, but for a while, I was running something that literally didn’t exist.
Steve was busy trying to put the company back together and create products people actually cared about again. That would ultimately become the iMac. And he wanted a mailing list to distribute information about this new product, and it needed to scale, so I got to build a new server just for apple to everyone announce lists with large expected audiences. And once the iMac was announced, so was imac_update, which was Apple’s first serious foray into using email as a marketing tool. Later on, we added in Apple News and iTunes Update, both of which still exist today.
Then things got crazy fun. In 1999 I proposed building a system that would send marketing emails to individuals, not via user subscription lists. At the time, Apple was using an outside vendor that cost about five cents an email, and since every email sent required cutting a check to pay for it, it didn’t get used very often. I felt we could bring it in house, save Apple some money and get better tracking data and a better quality email experience. I got the go-ahead, and we brought in a junior programmer to work with me on the project.
It took a few months, big hunks of perl and PHP code and some sendmail servers, but we got the beast running. Our original goal was to, we hoped, send out 10 million emails the first year.
We hit that goal in six weeks. And we now had a tiger by the tail, and it was off and running. Once they didn’t have to get every email campaign approved and paid for, they started finding ways to use email. This started a fundamental shift from Apple marketing primarily via paper mail and advertising to using e-mail as a primary tool along with the magazine and mass media advertising.
That was my life for the rest of my time at Apple: it was all about keeping that system running, making it better, and most importantly, scaling. I’m not joking when I say for the next 3 or 4 years, when we did our capacity planning we assumed massive increases in volume, and each year, our actual volume was at least 2X our we-thought-inflated projection.
We added support for foreign languages, and then for non-Roman languages like Japanese (a crazy thing to do in Perl 5, but it worked). After a couple of years imac_update retired, but the Apple eNews became a primary tool for sharing things to the Apple community. iTunes was released in 2001 and they wanted a mailing list for it as well, and popularity of that exploded, going out to tens of millions of readers every week — ultimately in 24 languages.
For those worrying about spam, these systems were very clean, and on a typical mailing, the unsubscribe rate was under half a percent of the sent emails, and spam complaints were usually in the noise. We made sure un-subscriptions were easy and obvious and that requests were honored. In the first year we sent out about 150 million emails, and it wasn’t long before we were doing that a quarter, and then a month.
Did we save Apple money? Definitely. Beyond that, we were creating income by encouraging people to buy from Apple instead of resellers — we did an analysis a few years in that showed about a $100 million gain in revenue.
Jobs recruited Sculley by asking him if he wanted a chance to change the world. This project turned out to be my opportunity to help reshape a company and “move the needle” in that company’s finances. How often can someone say they did that?
It was crazy fun but also stressful. I might spend a morning on a conference call with Apple UK and evenings in chat working with Apple Singapore. Announce days were crazy busy, of course, with things timed to go out as products were made public. At times it was too much stress — I came back a staff meeting where we celebrated my 15 year anniversary, went back to my office, and started crying — and more or less didn’t stop for six weeks. My team rallied around me, and I put things back together again (and spent some time in therapy figuring out how), and we kept moving it forward and kept it growing. We were big enough that when we sent out a blast, we saturated Mac.com’s incoming email; ultimately we setup a special connection to them to moderate the incoming traffic.
And so it went. I occasionally got involved in a special project, usually when Steve wanted something done in a timeframe that the usual IT systems weren’t nimble enough. One near the end of my time at Apple happened when Steve cut a deal to issue coupons to give buyers of the new Red Hot Chili Peppers album early access to tickets to an upcoming tour. I found myself in a meeting with about 15 people where the project was explained, and the main IT team explained it’d take about six months to implement. Steve wanted it in two weeks.
And I realized everyone was starting at me. So I thought it through, explained an approach and we agreed to try it. I had it running — 250 lines of perl code and a couple of database transfer points — in two days. If you got those tickets to the concert, well, you’re welcome!
After that meeting one of the product managers came up to me to ask “what are we going to do next time?” — it was already known that I was looking to at least exit the project and IT, although I hadn’t yet committed to leaving Apple. And my only response was “unfortunately, that’s no longer my problem”. In fact, Apple has done just, as I expected them to.
The last six months or so were pretty rough. I felt like the stress was tearing me apart, and I wasn’t always handling it well. I hurt some people who didn’t deserve it, and frayed relationships I cared a lot about. We finally had a significant systems failure and it took us 13 hours to get it back online — only the second big outage of the thing in its existence. What’s worse is we knew about the possibility and were about two weeks from implementing some changes that would have fully mitigated the stupid thing, but that’s life some days.
And after that, I walked into my directors office to update him on things, and I said to him “I think I’m done”. And we started planning transitions.
I ended up giving a months notice; my bosses negotiated that out to two months. I spent the next six weeks lecturing on the details of the system and training others in operation and maintenance; all of that was videotaped for future reference, too. (Whee!).
I also spent time looking at new options within Apple. Aperture had recently come out and with my interest in photography, I really tried to find a way into that team, and got nowhere. There are stories to be told about why the Aperture team was a black hole at the time, but I wouldn’t learn about them until way later, when I was at Palm (and that’s maybe a story for later, someday). There were a couple of possibilities in the iTunes team but they didn’t pan out either. In reality, I think I knew I needed a reset and I believe Apple was ready for that as well.
I felt like the stress was killing me, but in reality (and I wouldn’t figure it out for another two years) I was suffering from major sleep apnea, and I was chronically exhausted from poor sleep. I’ve often wondered how things might have been different if I’d figured that out earlier, but I didn’t.
Finally the day arrived, I cleared out my office, said my goodbyes, handing it my badge, and moved on to the next stage of my life. I left behind the tools I’d built that allowed Apple to communicate at scale to it’s user community and leverage it’s marketing abilities to much larger audiences than it ever had a capability of, and as I said earlier, how often are you able to make an impact at a company as an individual that makes a significant impact in that company’s future?
The last time I got an update on those systems was years ago, and they were sending out something on the order of 10 billion emails a year, a number I struggle to wrap my head around. My junior programmer is still at Apple and now senior architect. Some on my team are still there, others moved on, and Apple has thrived and become the industry powerhouse it is today, and in some ways it makes me feel good about that day I talked my way off the layoff list and decided to stay with Apple. In my relatively small ways, I was part of making Apple a success again.
My one regret: those last few months when I wasn’t functioning all that well. I screwed up some relationships I really cherished, and I hurt people who didn’t deserve it. After I left, I gave myself time and distance from Apple and those people, and now, I wish I’d tried to keep in better contact. That said, I also don’t feel it’s right for me to go barging back into their lives after all this time, but if you read this and want to reconnect — please look me up. I’d love to hear from you.
