Is Serverless Hard to Adopt?
Understand the simple measures to make your serverless adoption a success
Photo by Mikhail Nilov on Pexels
More than a year ago, hiking at some altitude in the September setting of Semmering, Austria, we were lost at a junction on our trail. As the signposts were inconclusive, I reached out to my mobile for clues. Amongst several Twitter (X) notifications, a reference to Paul Johnston’s article Learning Serverless (and why it is hard) just flashed past.
On a normal day, I would’ve read it immediately. But given the location and the responsibility to guide my family out of the woods, I wasn’t ready to sacrifice a scenic Semmering morning for serverless.
I read it on return, and in his article, Paul has courageously hit many nails straight on! I wanted to add my thoughts sooner but with the commitment to complete the book Serverless Development on AWS. Building Enterprise-Scale Serverless Solutions (O’Reilly, 2024), I am here a year later to borrow the hammer and hit a few nails of my own, to look at the issue from an overall perspective and not specifically on learning alone.
In this article, I will take you through some of the essential factors to become successful when developing applications using serverless technology — be it in commercial product development, building data-driven applications, AI, or machine learning.
Is Driving A Car Difficult?
Is driving a manual transmission car difficult?Is learning to drive a manual transmission car difficult?
The answer varies based on who you are. If you’ve been driving for a few years, your answer to the first question would be an affirmative no. Because, over the years, you have become affluent with all the maneuvers, and they come naturally to you without needing to think about them.
However, when you were a learner, your answer to the second question would’ve been a definite yes. When you learn to drive, there are several things to consider — actions that require coordination, activities that happen in a sequence, and observations to do in parallel. All these while you are in motion with the vehicle.
Your ability to turn the steering wheel alone won’t qualify you as a driver.
If you now consider an automatic transmission car, your activities as a driver are somewhat reduced as the mechanics of the car take care of certain actions (like how the cloud provider takes care of some of the heavy lifting in serverless).
Many things in our daily lives are challenging, but we learn the skills and follow the proven path to become better at them. Serverless development is no different. It may sound a bit easy (like an automatic transmission car), but it won’t make you start the engine and jet off instantly.
A tangled serverless implementation. Source author.
There are many ways to do things wrongly in technology — more so in serverless. Building a tangled event-driven architecture using serverless services is easy and takes no time. However, architecting and developing a modular, observable, and sustainable solution requires knowledge and the right skills. This doesn’t mean serverless is hard.
There are many ways to do things wrongly in technology — more so in serverless.
Serverless is hard if you don’t understand its ecosystem.
A car has many parts, from a high level to a granular level. The car engine is mainly written as a single unit but has several components. A functioning car needs a driver (ignoring the driverless one) and carries passengers. In a way, all these together form the ecosystem of a car.
Too often, many portray serverless as an architecture blueprint, Function as a Service (FaaS), or a framework. To me, it is beyond all these and more than what we usually imagine it is. Serverless, in a way, is a technology ecosystem. When you and I work with serverless, we also become part of it — similar to how the driver and passengers are part of the automobile ecosystem of a car.
The serverless ecosystem has many factors. Source author.
As shown in the picture, the serverless ecosystem has several factors. Serverless technology brings its unique characteristics. The cloud platform and its managed services form the foundation. Developmental practices, tools, and frameworks enable the flow to bring value faster. Business stakeholders collaborate with engineers to build modern capabilities on serverless for customers. All these and more are part of that ecosystem.
The purpose of the above portrayal is to make you think beyond writing functions or debating over an infrastructure tool. Think of the diversity Serverless brings to your organization, to your team, and to you, as you will be wearing multiple hats when you work with Serverless. Thoughts like the one below from Lee Gilmore are more regular than in the past as we mature and become part of the serverless ecosystem.
Serverless Advocate 🥑 on Twitter: “In the #Serverless World when we have little to no line between infra (IaC) and business logic (all just code), it feels like the role of many in the cross-functional team is now a ‘DevArch’ one.I’ve seen this change over the past few years, where devs need to know architecture / Twitter”
In the #Serverless World when we have little to no line between infra (IaC) and business logic (all just code), it feels like the role of many in the cross-functional team is now a ‘DevArch’ one.I’ve seen this change over the past few years, where devs need to know architecture
Serverless is hard if you start with the wrong mindset.
A couple of years ago, an engineer reached out for guidance on his serverless journey. A few minutes into the chat, it became clear that he wanted to implement the logic of his Lambda functions in a cloud-agnostic way so that his serverless applications could be readily deployed to a different cloud provider if and when the decision-makers of his organization wanted to switch.
Understanding his intention, I asked about his approach to the non-FaaS services and how he would make them cloud-agnostic. His explanation was some kind of grandeur implementation of the hexagonal architecture!
Toward the end of the call, I realized he had yet to deploy any serverless workload in production!
Though thoughts and approaches like the above please the board, the challenges and the value to the business are not assessed. Enterprises adopt technologies such as serverless to improve velocity, increase flow, and build a competitive advantage. Approaching serverless with the above mindset is like chasing a mirage and never getting anything done and delivered any time soon.
Engineers often point to the multi-cloud configurations offered by frameworks as an example (or excuse). The fact is tools provide such capabilities so that everyone can use them, and you don’t make your multi-cloud decision based on what a framework offers. When such wrong strategies fail to produce value, you hear cries – switching to serverless is hard, confusing, and counterintuitive — as Paul mentions in his article.
Development tools often support multi-cloud capabilities, but you don’t make your multi-cloud decision based on what a framework offers.
Serverless is hard if you think it is too easy.
I was invited to a meeting to listen in and approve a serverless architecture proposal. An engineer presented a well-crafted virtual board and walked through their architecture. Lambda functions dominated the solution, with a couple of DynamoDB tables scattered around. I began to feel uncomfortable, and apologetically interrupted the conversation and asked the question-
Have you built serverless solutions before?
And the engineer answered, Yes!
After asking a few more ‘Whys’ on some of the choices made in the design, the engineer admitted that previously, they had written Lambda functions for simple features and not architected nor built event-driven microservices.
My intention here is not to undermine the engineer but to highlight an important phase we all go through as we pick up and work with a new technology. Knowing how to write and operate Lambda functions is definitely the right start, but at the same time, you should not get carried away thinking everything is too easy in serverless. This attitude is what makes you implement tangled serverless applications.
I agree that for someone new to serverless, the knowledge of splitting the problem into microservices, identifying the synchronous and asynchronous parts of the application, thinking in terms of events, or designing for observability is complex. The fact is, many of these aspects are not new and not specific to serverless. The difference is, with the siloed team structures of the past, you, as a programmer, never got the exposure or were involved in the process.
When you learn to drive, you take several lessons before you feel confident for the practical driving test. Similarly, it is necessary that you assess your serverless skills and undergo the necessary learning (discussed in a later section) alongside your job as a serverless engineer.
Serverless is hard if you exclude engineers from architecture discussions.
An approach I follow while guiding teams is to assign the ownership of a service or feature development to one or more engineers — from idea to production. They become part of the journey from early conversations with the product teams, stakeholders, and technical experts to gather knowledge about the solution they will be building. They sit through necessary ideation sessions and problem analysis and start drafting the architecture with guidance from seniors and experts before bringing out a Solution Design Document for everyone to review. From here, implementation tickets get created and go into an iterative development and delivery phase.
The Significance Of Solution Design In Serverless Developments — Part I
The above approach is intentional. Though there are drawbacks and criticisms, it brings many benefits to the team and to engineers’ professional growth. To educate engineers and make them part of the serverless ecosystem, you must stop feeding them with architecture blueprints. Instead, you evolve the architecture with them by feeding ideas and resources to make them learn, develop, and deliver.
Don’t craft your serverless architectures at the architects’ ivory tower but create them in your team’s engine room.
Once in an organization, there was a cross-team collaboration to bring out a new feature. There were a few ideation sessions with a bunch of people before agreeing on a high-level proposal that was good enough to be detailed in a solution design. Later, a couple of engineers were assigned a ticket to implement their team’s part of the solution. However, these engineers were never part of the previous sessions, had no briefing, nor seen the virtual board that captured all the scribblings, drawings, and thoughts of the previous conversations.
As you can imagine, the above engineers were put in a difficult situation. Regardless of the ability of engineers and the triviality of the problem, make sure your serverless development does not start with Visual Studio (VS) Code. If you do, there is no guarantee that your serverless experience is going to be any smoother.
Regardless of the ability of engineers and the triviality of the problem, your serverless development does not start with VS Code.
Serverless is hard if you demand perfectionism over pragmatism.
In my talks on How to Grow Your Serverless Teams, I share a slide to show how a practical thinking team can accelerate with serverless while a purist-minded team that tries to make everything perfect with pessimistic views falls behind.
Two types of serverless adoption camps. Source author.
Often, information overload and perfectionism of ivory tower architects with no practical engine room experience can make your serverless adoption hard and your experience bad.
I once heard the story of two serverless teams during a community conversation. Both teams had commonalities. They operated inside their respective boundaries — i.e., the bounded contexts — developed event-driven microservices and had well-defined APIs.
One team owned and operated their microservices in a single production AWS account (with separate accounts for test, QA, pre-prod, etc.), kept their Lambda functions outside of custom VPC (Virtual Private Cloud) and configured VPCs only where necessary, and hosted their APIs on Amazon API Gateway with the necessary Usage Plans. Their services are coordinated by a custom event bus on Amazon EventBridge. For this team, it sounded effortless to develop, deploy, and operate a new microservice.
The second team, however, guided by a purist school of thought, chose to host each microservice in a separate AWS account and deployed their Lambda functions inside custom VPCs. In addition, all their APIs were hosted on a different gateway landing platform. With twenty and a growing number of microservices, they were struggling with the complexity involved.
Of course, there are situations when we make different choices and decisions, but we should evaluate and course correct to avoid falling into an unmanageable and no-going-back situation.
One of the primary motivations for adopting serverless is to offload the heavy lifting to the cloud provider, but if we go against serverless’ strengths with our strategies, serverless can only get harder.
Serverless is hard if you corrupt team boundaries.
When it comes to team boundaries, everyone thinks of the Bounded Context (as in Domain-Driven Design). However, other boundaries are relevant to autonomous two-pizza serverless teams, as listed below. Corrupting boundaries causes consequences that make your serverless experience difficult.
Bounded context boundaryTeam responsibility boundaryTeam ownership boundarySource code repository boundaryCloud account boundary
The first three are dependent and overlap. If your team is the custodian of a bounded context part of an organizational domain, you are in a great place. In this case, your team is responsible for everything inside this bounded context boundary and owns all the features, services, and implementation artifacts.
All your implementation artifacts for the assets you own are in a repository your team members contribute and share among themselves.
The AWS cloud account boundary marks the operational boundary within which you deploy and operate your cloud and serverless assets.
In an ideal world, you will have a one-to-one mapping between your team and these boundaries, as highlighted below.
Your team exists because of the bounded context.Your team is responsible for and owns everything within the bounded context.Your team has a repository to which your team alone contributes.Your team operates everything it owns in its dedicated cloud account.
Your life as a serverless engineer is as simple and enjoyable as it can be!
Now, let’s start loosening up and open the boundaries, and imagine what might happen.
Say, due to business priorities and to meet delivery expectations, you bring engineers from another team to share the responsibility of contributing to your team’s boundary, forcing your team’s focus areas to change.
How do you make sure the teams know each others’ ways of working?How do you make sure the code quality is uniform?How do you make sure the selection of AWS services and architectural patterns align?How do you make sure there are no gaps in the operational responsibilities?
These teams can collaborate to address and ease the situation. But collaboration means more chats, pairing, reviews, discussions, meetings, etc., and these will impact both teams’ focus and efficiency.
Open team boundaries and shared responsibilities without adequate thought can have a negative impact on an organization. Serverless development can get harder, along with frustrating developer experience.
Serverless is hard if you don’t invest in people’s growth.
The world around us is filled with shortcuts. Several shortcut avenues teach you to write Lambda functions. This fast-food style of learning is good only up to and until the food lasts. To sustain your appetite, you need to keep ordering more.
To be successful in serverless, you must keep learning.
Paul’s article focuses mainly on learning and how people who enter serverless often struggle to get grips with the technology and its ecosystem. As Paul describes, for someone new to serverless, even the concept of single-purpose Lambda functions itself takes time to digest.
To be successful in serverless, you need to take many strides once you get to grips with single-purpose Lambda functions. Several areas act as stepping stones for you to progress towards a successful destination that requires many hops, and you gather many traits along the way.
I often highlight how serverless brings engineering diversity to teams. This realization is essential if you want to make serverless easy.
Engineers should be guided, nurtured, mentored, or trained (or whatever the terminology is used in your organization) to make them understand stakeholder requirements, allow them to propose architectures, instill in them the benefits of single-purpose functions and microservices, show them how to incorporate security, teach them the observability principles, let them deploy to production, and tag them as owners of their services. These won’t come overnight or from watching a few videos on YouTube. This is where quality training and a long-term people growth strategy come in.
Traditional siloed specialists versus diverse serverless engineers. Source author.
Try not to restrain engineers with corporate bureaucracy. Allow them the freedom to learn new things, attend conferences, and be part of technical workshops and collaborative activities.
I was chatting to an engineering manager at a conference. She opined that workshops such as EventStorming and Architecture Katas that span for a couple of hours or more are a waste of time and affect the team’s productivity. So I asked her how she would handle productivity if a couple of opportunity-denied and disgruntled engineers called in sick for a day or two. She didn’t have an answer!
Serverless boot camps are a simple way to equip engineers with the basics of the serverless technology ecosystem. Some organizations have successfully implemented such programs. Matt Coulter, an AWS Hero, once mentioned a successful program for new starters in place at Liberty Mutual. Organizations often deprioritise such initiatives citing budget constraints without even assessing the benefits they bring in.
One of the challenges when it comes to serverless training is the quality and coverage of technical, developmental, architectural, and operational elements in the syllabus. Amongst several courses, I have heard great feedback on Yan Cui’s Production-Ready Serverless training workshop. Yan is an AWS Serverless Hero and a powerhouse of serverless knowledge.
Make engineers understand stakeholder requirements, allow them to propose architectures, instill in them the benefits of single-purpose functions and microservices, show them how to incorporate security, teach them the observability principles, let them deploy to production, and tag them as owners of their services.
Serverless will become easier…
Like how driving gets better and easier with the hours behind the wheel, serverless becomes productive and enjoyable as you accumulate experience and become comfortable with its ecosystem.
Rollercoasters are not everyone’s favorite. The most common advice thrill seekers offer to others is — You just gotta learn to let go!
You must learn to trust serverless technology to capitalize on the undifferentiated heavy lifting it offers. If AWS is offering managed solutions, make use of them to multiply business value. Why work against and build complex solutions you will never need?
Get inspiration from those who have successfully adopted serverless.
To close, I couldn’t find a better phrase than the theme of Momento’s AWS re: Invent 2023 Community Party–
Believe in Serverless!
Yes, believe in serverless and learn to let go!!
Is Serverless Hard To Adopt? was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.
%%
Also visit my page; https://rakily.com/website-list-1460/