So long, Amazon
By Dave Allie
Published Jul 30, 2021 • 9 min read
Published Jul 30, 2021 • 9 min read
This month was my last at Amazon. The last two years of building software at mind-boggling scales for Amazon customers
around the world has been engaging, challenging and insightful. I’m sad to have hung up my badge and walked out the
doors for the last time, but thankful for the knowledge I’ve picked up and the advice I’ve received along the way.
I thought I’d take the time to reflect on some of the best and worst parts of life at Amazon as an engineer. While
it doesn’t come at all close to a comprehensive list, and may differ greatly from other people’s ups and downs, these
are the aspects that resonated with me or irked me during my time there.
Before starting at Amazon 2 years ago, I had some preconceived notions of how Amazon built, launched, and maintained
software. I was picturing a slow, lumbering giant, filled with many small, and easily forgotten cogs. With the overall
machine struggling to course-correct due to its own inbuilt inertia. However, throughout the interview process, and in
my first few days, I was completely purged of my delusions.
Amazon splits high-level business focuses into separate areas, each with its own clear and distinct owners. This
split of responsibility is repeated again and again until you’re left with an atomic product owned by a product manager.
For the majority of my time at Amazon, I was working in the grocery space, focusing on the technology and experience
of substitutions for a customer’s items. This split of responsibility and clear ownership meant that our team’s output
had a single point of approval, our PM. There was always a single person responsible for ensuring that what we were
building was moving us towards our end goal and any deviations in our projects could be approved without escalating all
way up to Jeff.
This is the true representation of decentralised decision-making in practice. Ownership is effectively delegated and
owners are empowered to independently make decisions on behalf of their product space. As an engineer, it was extremely
relieving that in a company as big as Amazon, I only needed to convince my team and one or two other people in order to
get things done. I have previously witnessed attempts at decentralised decision-making where owners attempt to involve
everyone in all decisions and delegate ’decision owners’, but still want a strongly weighted voice in the decision. This
always resulted in huge amounts of confusion around ownership, and a risk of accidental undermining.
The availability of extremely smart, and helpful people at Amazon was amazing. I had direct lines of communication to
people with an insane amount of experience across the whole software development lifecycle, with expertise in just about
every area. There’s bound to be at least a few experts at your company or within your development team, wherever you
work. The Encyclopedia Britannicas of the tech, tooling, and processes you use on a daily basis. The people you go to as
a last resort once every other well of information has come up dry, who always seem to have the answer. Now imagine
expanding your team to hundreds of thousands of people, and scale the number of experts as well. That’s what it’s like
working at Amazon. If you have a question, about literally anything, chances are you can find an expert at Amazon, and
they’re just a DM or an email away.
Amazon’s Leadership Principles are thoughtfully considered and unironically embodied
by everyone within the company. They aren’t just plastered to the wall and paid lip service. They reflect both the
values of the employees and the company. It could just be brainwashing, but I truly believe that the leadership
principles are useful core tenants to lean on in all parts of Amazon. They are used during 360° reviews, hiring,
promoting, and are considered during almost every critical decision point. Including system architecture designs,
launch decisions, and feature prioritisation. Having a common framework for backing up thoughts and decisions which
represent the best outcome for the company meant that disputes were infrequent, and when they did come up, you were
assured that the other side of the argument was trying to fulfill the same criteria as you.
While there was a lot of great things at Amazon, there were some inescapable not so great things about working there
too.
Leaving money on the table is never a good feeling, especially when a large portion of that money comes in the form of
ever-growing Amazon stock. Amazon has perfected the
process of fitting golden handcuffs to their employees, through a combination of a high base, and a generous amount of
stock units. It’s pretty clear how the base component works, they just pay equal or more than most employers in the
cities they operate (at least for corporate employees 😓), the catch comes with the stock units. Amazon awards the
stock on a rolling schedule years before they vest. So if you do decide to leave, you’re going to be walking away from a
few years worth of promised and pre-awarded stock units. Personally, I found it a tough pill to swallow, but I wasn’t
dependent on those future earnings, so I just stopped considering un-vested units “mine”, and it became clear that the
only thing I was losing from quitting was future potential and nothing with value today.
Amazon has an unbelievable number of internal tools, services, and frameworks. They have their own internal version of
StackOverflow, internal wikis, internal video hosting platform, custom-built Git repo hosting and code exploring, CI/CD
platform, deployment services, handfuls of different design systems and component libraries, internal URL shortener, the
list goes on. If you can picture a public SaaS product, Amazon most likely has an internally built version that does
the same thing. Unfortunately, if the tool or framework you’re looking at isn’t on the critical path for a majority of
engineers at Amazon, then you’re probably picking up a deprecated, scheduled to be sunset, or completely undocumented
piece of tech. In my experience, Amazon’s external documentation isn’t fantastic, but it’s consistent and serviceable,
the same can’t be said about all internal documentation.
You’re as likely to find a completely documented tool as you are to find a completely undocumented one. It’s infuriating
typing the same queries again and again into the internal StackOverflow or wiki searches to discover not a single shred
of information or replication of your issues. Often I would end up resorting to searching through other team’s code to
find out how they’re using the tool and following up with DMs and emails to get my own use-case working. Once everything
is working, there’s no incentive beyond goodwill to help backfill the documentation for other teams, so the issue never
self-corrects. Even in my last week, I was receiving DMs from an engineer in another country asking for help
troubleshooting a specific issue with implementing an internal framework (which I didn’t own), just because I had set
it up for one of my team’s services over a year prior.
Because there’s no external funding for these tools, if you’re not able to prove that fixing bugs, making security
patches, or adding features to the tool is adding value to the company (often indirectly through productivity
improvements of other teams), then it’s likely that you’ll have to put that tool into KTLO (keep the lights on) or
sunset it completely. One day you’ll open a tool, and there’ll be a big red banner at the top of the screen with a date,
threatening to turn it off unless some other team has the courage to take up the mantel of ownership. While this does
result in almost a Darwinian system of tools where only the best and most-used survive, it also has the tendency to
catch extremely useful, but underused tools in the crossfire.
To get around this, some teams have procedures in place to encourage other teams to commit fractions of their time to
improving tools in exchange for a few benefits. The most extreme example of this I saw was an internal design system
that had a “Standard” and “Premium” tier complete with a comparison table showing the differences between them. There
was no difference between the actual design system or components between the tiers, the differentiators were things like
access to a private Slack channel with the authoring team, influence on the roadmap, and workshops. While it is possible
for teams to charge each other internally for use of their services, or tools, it is often reserved for situations where
usage is directly tied to an external cost (for example sending SMS). Instead, if you want access to the Premium tier of
this design system, you had to commit an engineer on your team to help build out part of it, reducing the load on the
owners. I had a chuckle when I first found the tiers, but after thinking on it more, it troubled me that there needed to
be such backroom deals to keep important tools off life support.
Absolutely! I had an amazing time at Amazon, and learnt a lot about what it takes to operate within one of the
biggest companies in the world. My time there was invaluable and granted me a significantly different perspective from
my previous roles, where I could remember the name of everyone in the company. I’m truly grateful for the opportunity I
was given.
I’m promoting myself to customer and am off to tackle a new challenge! I’m joining a founding team to help solve big
problems in the construction sector with new software. If that sounds interesting, and you’d like to chat more about it,
feel free to reach out on LinkedIn or drop me an
email.
I want to take this opportunity to thank profusely the 3 managers I had, and 2 teams I was able to be a part of, during
my time at Amazon. All of you are awesome, and hopefully we can work together in the future. 😍