What Adrian Did Next — Part 3 — eBay — 2004 to 2007
I’d left Sun (part 2 in this series), and had a few months off over the summer, so (of course) got married to @laurelco, bought a “fixer upper” house in the Los Gatos mountains, and worked on getting it tidied up.
From a career point of view, I was looking for a change and decided to get out of the business of selling computers to big companies and figure out consumer oriented technology at eBay. I’d helped eBay recover from capacity related outages in 1999 and had setup their capacity planning processes. I had also turned down a job offer at that time, and when I called back in 2004 they took me on as a Distinguished Engineer in their Operations Architecture team.
At the time eBay was one of the largest online businesses and was developing patterns that later became widely adopted. Their operations and deployment model was highly automated, pattern based, and horizontally scaled. Their database model later become known as NoSQL, although it was implemented on top of Oracle, each database held one table and indexes, and there were many sharded Oracle databases for each data set so it could be scaled horizontally as well. I learned a lot about building, operating and optimizing large scale global web services and made some good long term friends including Tom DiGrazia, Jeremy Edberg, and Randy Shoup.
After a year or so, and a re-organization, eBay Research labs was formed by Eric Billingsley, and I joined to help recruit the team, including a good friend from Sun, Paul Strong, and build out the innovation programs. It was a great opportunity to explore new technologies and broaden my experience. We visited Xerox Parc and IBM Santa Theresa labs to meet with their leaders to get some ideas on what worked and what to avoid. We setup an innovation program to gather ideas from employees — who kept suggesting things that already existed — and ran advanced development projects. I got interested in programming mobile phones, and at the time, in 2005, the built-in email app on the Blackberry was state-of-the-art. We found one fairly advanced Nokia phone and built a flash based instant-deals app for it. I had to argue with people that it was a research project, so it didn’t have to be deployable at scale, and it ended up looking cool and showing signs of the addictive/immersive nature of apps to come. That app concept was re-engineered and released as an eBay iPhone app many years later.
The eBay/PayPal duo was joined by Skype while I was working in the labs, and I turned my attention to figuring out how Skype worked and researching ways of leveraging it. My friend and colleague Josep Ferrandiz was working on capacity planning for PayPal, and came up with the idea of a peer-to-peer marketplace, that would scale with the end user nodes, the way Skype did, so we wouldn’t need to bulk up datacenter capacity for peak loads. We prototyped and patented Skype Exchange, a marketplace (like eBay, but actually more like Craigslist), that used the Skype extensions API and transport as its platform (it transacted between Skype accounts), and which monetized via PayPal (you could PayPal money to a Skype account). We thought it was a neat idea, but one reaction from management was “We don’t want people to think we bought Skype to do that…” and we didn’t get support to productize it. My own theory for why companies kept buying Skype every few years was to to prevent Google from buying it… it was sold again and ended up at Microsoft.
At one point I was figuring out how to prototype a complex peer to peer multi-stage offer/bid/accept/pay transaction protocol and discovered that there was a business process standard (BPEL) based on Pi-Calculus, which I’d learned about working on Occam in the 1980’s, and decided to build a simulator using a port of the Occam language that I found was being maintained at the University of Kent. I remember telling Eric “I just wrote 1000 lines of code in a language no-one’s really used in 25 years to build this actor based simulator, and it works!”, and he thought it was cool. I wrote it up as an academic paper, presented it at an IEEE conference, and along with a few patents, that’s all that came out of the work. Many years later I re-wrote the simulator in Go and had some more fun with the concepts at Gophercon 2016.
I also found out the main problem with research labs, which is that they create an us-and-them dynamic in a company, where teams outside the lab felt excluded, and when we wanted to productize ideas, they were seen as needing too much re-work to make them production ready. There were some good successful projects, like a new scalable search engine with real-time index updates, but it became frustrating. In addition, early in 2005 eBay, management decided to put up prices for sellers which drove an exodus to Amazon, and had also spent a lots of money buying adverts on Google (eBay was one of the earliest large revenue sources for Google) helping Google to become the new hot stock. The eBay stock price went from growing fast to going no-where, and the options I got when I joined were under-water. It was early 2007, and time to move on again.
I talked to a few startups, and interviewed at Google, but they were fixated on repeating the operations job I had been doing before, not the job I wanted to do next. Then I got a call from Jessica Neal, who was recruiting for Netflix (and is currently their Chief Talent Officer), and decided to see what opportunities they had open. At the time Netflix was small and struggling, it’s DVD rental business under siege from Blockbuster Total Access, and they were about to launch the first streaming service, that ran on Windows PCs only, and had a very small content catalog. They weren’t known as a technical leader apart from the movie recommendation engine competition The Netflix Prize, which had launched recently. They were looking for people who could help them scale their infrastructure, and hired several people from eBay (a few miles up the road) around that time. The interview process was interesting, and their description of their corporate culture was fascinating (this is years before the “culture deck” was published in 2009). I joined in part to see if they really did run the company that way, in part because they wanted me to be a director for an engineering development team, which I hadn’t done before, in part to learn how personalization algorithms work, and finally, because they were prepared to pay well to get a “distinguished” scalability expert on their team. The whole company was a few hundred people. They convinced me that they had a better chance of succeeding than the markets thought (their split adjusted share price was about $3 then, and is hundreds of dollars now), and I would learn a lot, which I did. In fact, for the rest of my career, I’ve been telling stories about how we did things at Netflix between 2007 and when I left in 2014. That’s going to take more than one blog post to summarize…
