In my career spanning over 22 years as a technology consultant, I have collaborated with a diverse array of organizations, ranging from public transportation enterprises to financial institutions and healthcare technology firms. Working with complex tech and architectural problems in these areas has really improved my skill in spotting recurring patterns – or perhaps more accurately, anti-patterns – of inefficiency. These patterns often appear as persistent operational bottlenecks, redundant processes, or suboptimal use of technology, and they are blocking the ability to implement processes to constantly improve. 

In this article, I share my experiences from greenfielding a tech organization, where we had the opportunity to start from scratch, aiming to avoid some of the key challenges prevalent in the IT industry. First, let's have a look at some of these challenges.

Some key challenges of modern tech development

1. Business goals not aligned with product development

Numerous organizations have more or less well-defined and explicit business goals, yet often grapple with the challenges of structuring their development and operational processes to effectively achieve these objectives. Several years ago, I was involved in a project as a solutions architect for a major banking institution. This project was situated in the initial stages of developing a payment solution product.

In the early stages of this project, my main job was to help the bank figure out how to turn their business goals into a real, working tech product. Different people involved in the project had their own ideas about what the product should do, but they didn't have the ability to see how these ideas fit with the client's overall goals. This confusion led to a lot of doubt among everyone working on the project. People started to worry whether the product would be good enough to compete in the market.

2. Shortfall in professional competence

In my time working in tech, I've seen one thing come up over and over again, and I think it's a huge issue. I've been in so many meetings where clients boast, "We've got the best people in the business." But from what I've seen, it's pretty rare to actually come across "the best people." Yeah, I've worked with some of those open-source savants who used to be the team tech company's superstars 15 years ago. But the way we build IT systems today? It's a whole different ball game. What we really need now is the right people, not just those labeled as "the best."

I think that modern technology has been moving faster than most companies can keep up with. Things like cloud computing, data mesh, serverless systems, IoT (Internet of Things), and DevOps have brought lots of new chances to do things differently, but they've also made everything in system development more complicated. Ten years ago, it was easier to figure out what kind of people and skills a company needed to have a good tech team. The process of hiring people wasn't as complicated and didn't need specialisation at the same level as today.

As a consultant, I've been in numerous meetings with various clients in the tech sector. These experiences have convinced me that one of the industry's key challenges is developing the ability to identify the right talent to bring into a company. Failing to recruit the appropriate specialists can lead to substantial costs, often manifesting as technical debt. I frequently observe this kind of debt being created in real-time due to architectural decisions that don't align with a project's overall objectives, or from choosing technologies based simply on familiarity. Countless times, I've questioned the logic behind certain technological choices, only to hear, "That's what the developers were accustomed to." This approach, especially in cloud computing, can lead to significant and unnecessary expenses due to a lack of proper knowledge and experience.

I strongly believe that for an organization to stay competitive in tomorrow's market, it must have the right people at the right time. In an increasingly competitive market, finding the right talent often requires a global search. Companies that insist on having employees in the office at all costs are already falling behind their competitors. Those who have embraced efficient remote working processes have a distinct advantage because they can attract and retain the right talent for the future, without the restraints of people being in a specific location.

From my experience, the optimal solution is a blend of specialized consultants, proficient in specific fields and technologies, and in-house employees who understand the company culture and exhibit strong ownership of their work. This combination offers both the deep expertise and the committed, culturally aligned workforce needed for long-term success.

Companies that insist on having employees in the office at all costs are already falling behind their competitors.

3. Lack of autonomy

In the early stages of my career, I was a developer on a project for a major Norwegian public sector organization. This project was managed by a larger consulting agency, which effectively directed all operations. The project structure included 7-8 substantial development teams, alongside a dedicated team of architects. Each development team was tasked with implementing a specific feature of the system, functioning similarly to product teams. Each team was led by a team leader, interestingly, none of whom had a background in technology. Throughout this project, I cannot recall that I met any of the product owners or members of the management team.

The development process was structured in such a way that the architect team, in collaboration with the product owners, would define Jira epics and tasks, and then allocate these tasks to different development teams based on system features. However, this approach led to a situation where no team had a clear understanding of what the others were working on. It seemed the underlying assumption was that the architects, being a sizeable and skilled group, would meticulously craft the Jira tasks so accurately that they would seamlessly integrate when translated into code.

Of course it did not seamlessly integrate... The tasks didn't fit together as smoothly as anticipated. We frequently encountered bugs, necessitating continuous reporting in Jira. This, in turn, forced the architect team to repeatedly send feature branches back to the backlog for re-evaluation and re-architecture. Such scenarios often required a complete redo of the entire process for a single epic, causing significant delays in the development of features. Management seemed to harbor the belief that keeping the architects and developers separate was a strategic move. However, this separation only contributed to inefficiencies and disconnects in the workflow, hindering the overall development process.

If I would find myself in a similar situation today, I would step up to the project management and ask what the hell they were thinking and gently suggest they reassess their entire operation. However, back then, my role was limited to observing and being part of the entire gong show. Reflecting on my career, I've realized that granting autonomy to tech teams is often a highly effective strategy. This approach typically leads to more innovation, motivation, and ultimately, success in the projects I've been involved with.

4. Failure to understand that there is a problem

Failing to grasp the bigger picture can lead to an unawareness of inefficiencies in your project or even within the entire company. This issue can present itself in various forms. On one hand, there might be complex technical solutions that are challenging to comprehend without specialized expertise. On the other hand, there could be more high-level structural issues that require a deep understanding of organizational dynamics and strategy. These problems may not be immediately obvious but can significantly impede progress and productivity. For instance, there could be inefficient communication channels or bureaucratic processes that slow down decision-making and innovation. Without a clear view of these larger organizational issues, it's difficult to implement effective solutions, leading to prolonged challenges and reduced operational efficiency. Recognizing and addressing high-level, structural issues is as crucial as solving the technical ones for the overall health and success of a company.

During an assignment as a cloud specialist, I had a conversation with a project manager. This individual did not have a background in technology, let alone a deep understanding of cloud computing. We were discussing whether the company should transition its tech infrastructure to a public cloud platform or continue with an on-premise solution. At a certain point, I explained to the project manager that given the company's goals for scaling and growth, migrating to the cloud later might be extremely challenging, if not practically impossible. To gauge their perspective, I asked, "Do you believe me when I say that this could be the case?" The project manager's response was straightforward: "No, I don't."

This situation showed that some (key) people in the company didn't really see the big picture or understand how important certain choices were at that time in the company's growth. We were talking about a decision that could really affect how well the company could work and grow in the future. However, it looked like this important matter was being overlooked by several people in the organization. I've learned that when you have more freedom to make decisions in a company, it's really important for everyone to understand what issues we're trying to solve, no matter what they are.

5. Cost optimization and sustainability

Controlling costs is crucial, and in my experience, most organizations I've worked with have had significant room for improvement in managing their production expenses. This doesn't just include direct costs from infrastructure and third-party software but also extends to manpower, office space, data, logging, and more. Specifically, I believe the following are among the main drivers of costs:

• High turnover rate
  Investing in employee satisfaction and retention can significantly lower the costs related to recruiting and training new personnel. I firmly believe that employees need to be given ownership over their tasks and should be empowered to make decisions that affect the final product. The people who actively seek such responsibilities are the ones you want on your team.
  
  
• Inefficient use of cloud computing
  Focusing on cloud cost optimization can potentially reduce cloud expenses by up to 40%, in my experience. I view cloud cost optimization as "low hanging fruit" in terms of reducing a company's production costs. Opting for multi-cloud or hybrid on-premises/cloud solutions is typically more costly compared to a cloud-native approach. Efficient resource utilization means less power consumption, equals less impact on the environment.
  
  
• Remote work and travel expenses
  I've always found it surprising that, despite the availability of technology for remote work, many organizations continue to allocate substantial funds for office spaces and employee travel. The environmental impact of not transporting people is substantial, according to research.
  
  
• Utilize analytics for decision making
  Many companies incur significant expenses due to the inability to make informed decisions regarding aspects like product development strategies and infrastructure optimization. Merely collecting data isn't sufficient; it's crucial that this data is highly accessible throughout the organization.
  
  
• Employees vs consultants
  Employ consultants as specialists for defined, time-limited projects, enabling internal teams to concentrate on core business activities. However, I'm cautious about using consultants to plug temporary resource gaps in daily operations. I have seen this approach becoming a crutch, preventing companies from addressing these gaps permanently.

Greenfielding a tech organization

A couple of years ago, I had the chance to collaborate with a client on a project that began with a clean slate, coupled with the ambitious mandate of doing everything "right." While perfection is obviously not possible, I appreciated the high level of ambition behind this goal. Armed with a compact team of highly experienced individuals, we were well-equipped to avoid many of the issues I had experienced in past projects. To start, we established a set of key requirements.

• Autonomous teams
  The company's product lineup demanded highly specialized product teams, comprising not just tech experts, but also professionals from various other fields. Our aim was to structure these teams to function with as much autonomy as possible.
  
  
• No key people
  The development processes must be sustainable and not dependent on a few key individuals, be they managers or developers.
  
  
• Get the right people at any cost (almost)
  Recruiting talent globally, regardless of their location, and structuring work processes to accommodate remote working conditions.
  
  
• Make the whole organization data driven
  Implementing the Data-as-a-Product concept to facilitate the sharing of data and metadata across various domains within the project. Looking into building a Data Mesh in the future.
  
  
• Build a continuous improvement culture
  Empower every stakeholder to take ownership and accountability for their respective segments in each project or product. Establish a flat organizational structure that encourages everyone to question and challenge decisions.
  
  
• Product development must be driven by tech teams, not by management
  To build tech products that are competitive in the market, it's essential to understand the complexities involved in their development.
  
  
• Team interoperability
  Every team should have the capability to function effectively across different domains within the company.
  
  
• Focus on cost optimization and sustainability
  Ensuring that the entire organization is conscious of the costs and environmental impact of its daily operations is crucial. In my view, waste, of any type and form, is fundamentally opposed to the principles of continuous improvement.

Building the team structure

Our criteria for team composition necessitated a blend of professionals capable of navigating all aspects of the organization's technical operations. This required putting together teams with expertise in:

• Platform and cloud architecture
• Kafka
• Kubernetes
• CI/CD
• Terraform
• Document databases
• Programming languages like Go or Python

Additionally, we included a functional product specialist in each team. By equipping teams to operate across various domains, we ensured that operations could continue smoothly, even in situations where a team might be missing certain specific expertise. Rather than forming separate teams for platform, security, operations, etc., we chose to distribute these competencies among our teams, addressing any challenges that arose from this integrated approach, by implementing necessary processes for knowledge sharing and interoperability between different domains.

Each team managed its own dedicated AWS development account independently, ensuring full autonomy. By using Confluent Cloud, we integrated tools like Sentinel and established processes to guarantee adherence to company standards, cost-efficiency, and robust security measures in our production environment. Platform specialists from each team, accompanied by an additional member for the purpose of knowledge sharing, had weekly meetings to synchronize on platform-related subjects.

Refer to the diagram below for a visual representation of our team structure. Note, this drawing does not give the complete overview of all relationships between teams, domains, tech and processes, but it draws the big picture, for illustration purposes.

Getting the right people

As I previously highlighted, securing the right talent is essential for success, a sentiment that I'm sure many managers and business owners share. Based on my experience, the most effective team dynamic is as follows:

• Tech teams should mainly be made up of full-time employees, in my opinion. This way, you get a team that really knows the ins and outs of your company’s tech and vibe. Full-time employees help create a tight-knit team that's totally on board with where the company's headed. Plus, they get to know the business really well over time, and there's a sense of stability that's I believe is important, especially for long-term processes and planning out the big picture.
  
  
• You need a mix of senior and junior profiles. There are research proving that diverse teams, in terms of experience levels, tend to have a more dynamic working environment. This can lead to more creative problem-solving and a more robust team culture.
  
  
• Specialist expertise is often required for short-term, exceptional scenarios rather than ongoing needs within a company. Typically, specialists are brought in as consultants to address specific challenges. I believe that their role should extend beyond just advising or solving a particular issue; they should also contribute to training the in-house teams, enhancing specific skill sets within the organization.
  
  
• In the process of hiring employees or consultants, I would advise to focus on their skills and experience – aspects that can be pragmatically assessed. Venturing into evaluations of personality, especially when based on a limited number of interactions or, less reliably, on personality tests, can be problematic. This approach risks overlooking highly qualified individuals who possess the necessary skills for the role. Personally, I prioritize hiring individuals based on their experience and skills, rather than basing my decision on my personal liking for them (which I don't trust that much).

Build teams that can operate remotely!

I can't stress enough how vital it is for future competitiveness in tech to run operations without being constrained by location and time. In Norway, where our company operates from, I've noticed a tendency among managers to resist this trend. They often prioritize hiring staff who are close to the head office over those with the necessary experience and skills. This issue, as I've previously discussed in this article, stems from a deeper problem: a failure to even recognize that there is a problem. This lack of awareness is a structural issue that needs addressing for true progress. Failing to recognize the link between inefficiencies in development and operational processes and the absence of appropriate expertise will hinder your ability to stay competitive in the evolving tech landscape. I stand by this statement 100%!

During the pandemic, companies switched to remote work almost over night, and those who did it right, reported the same or higher levels or productivity. A study conducted by Mercer, a human resources and workplace benefits consulting firm, surveyed 800 employers and found that 94% of them reported that productivity was the same or higher than before the pandemic, even with employees working remotely. The shift to remote work challenged the historical perception that employees need to be physically seen to be productive.

Ensure data and information flow within the organization

In every organization I've worked with, inefficient communication has been the primary problem, without exception. While countless books and articles delve into this topic, and I'm not an expert, the consequences of poor communication are clear from my experience. I won't dig deeply into the complex subject of interpersonal communication, as this is outside my expertise. However, for our greenfield project, we faced a very practical challenge: ensuring that data produced by our autonomous teams was accessible across different organizational domains. Solving this was key to maximizing their efficiency.

By applying the Data as a Product concept to separate data from logic, we facilitated a seamless flow of information across all domains. We utilized Kafka to store and provide data to all teams, opting for event streaming patterns over traditional APIs.

While this illustration may not capture every detail, it does the job of explaining the concept. Typically, a Data as a Product (DaaP) is a microservice, like an AWS Lambda or similar, responsible for producing data products. In our setup, we channeled these data products into Kafka topics, making them accessible to all product teams. However, not all data products originated from these teams. We also employed DaaP for distributing essential company-wide configurations, like Sentinel policies, which were rather seamlessly integrated into each team’s CI/CD pipelines. Whenever new policies were available, these were picked up by consumers and triggered events to update the pipelines. This is an example of how a data-driven approach can drive technological development within an organization, a trend I believe will become increasingly common in the future.

Implementing a data-driven approach involves more than just integrating an event streaming pattern in your architecture and sharing data across domains. It also means making decisions based on data. While most organizations use data to some extent, the real question is about the extent and frequency of its use. Having a sales manager report monthly sales figures to the CEO in a PowerPoint doesn't truly embody being data-driven. However, providing these sales numbers to product development teams multiple times a day, enabling them to identify patterns and adjust their development processes accordingly, is a step towards truly embracing a data-driven approach.

Enable teams to drive product development

This approach is becoming more common in the tech industry, particularly with startups and agile organizations. Adopting flat or minimally hierarchical structures to foster innovation, agility, and collaboration can be crucial for developing cutting-edge products. However, this isn't a new idea. I recall discussing similar concepts 15 years ago. But what really makes it successful?

Nowadays, most IT products need professional expertise that goes beyond what most IT engineers and architects know. We aimed to integrate this expertise directly within the teams, at the heart of product development. We moved away from the traditional product owner role, which often exists outside the product teams and operates at a more functional, meta-level in many companies. Including product specialists in our teams led to tech staff gaining a broader understanding of the overall objectives of our projects and the overall business goal, while functional experts developed a better grasp of the technical aspects of our work.

From what I've seen, having top management in the company too involved in making products can cause big and unnecessary delays. Such bottlenecks usually arise from a lack of appropriate personnel or ineffective organizational structures. I recall working on projects in the banking sector during a period when banks viewed IT only as a necessary expense, not seeing the potential for innovation in technology. At that time, it seemed to me that bank management considered developers and architects merely as trained monkeys, rather than contributors to product development.

Interestingly, many of the online banking products we use today were pioneered by tech-savvy individuals within the banks themselves. I often wonder how much more advanced these products could have been if the tech teams had been given greater influence a decade earlier.

It is crucial for every organization to critically assess their operations and question whether they are repeating the mistakes of the banking sector in the past. This reflection is key to fostering innovation and avoiding outdated approaches to technology development.

It is crucial for every organization to critically assess their operations and question whether they are repeating the mistakes of the banking sector in the past.

Cloud nativeness and optimization

With over a decade of experience in cloud computing and architecture, I firmly believe that operating IT operations on a public cloud platform, like AWS, Azure, or GCP (among a few other options), is essential for staying competitive in today's tech market. The reasons, which are too extensive for this article, include crucial factors such as scalability, operational cost control, faster time-to-market, and accessibility. These are so vital that most organizations can't afford to ignore them. Fully adopting a cloud-native approach, leveraging managed services, serverless technologies, and the scalability offered by cloud providers, we could easily foster cost and resource optimization awareness within the company, both financially and environmentally.

Our optimization efforts extended beyond just cloud computing and operational aspects; they also encompassed how individuals structured their workdays, meeting conduct, data sharing, communication, etc. In designing and implementing any process, we always factored in cost and sustainability, ensuring these processes aligned with these values. One such initiative was waste reduction in the development process, which involved identifying and eliminating unnecessary elements, such as redundant code or unused resources, to optimize efficiency.

Other cost-optimizing measures we implemented included adopting a DevOps culture across all teams, streamlining and automating deployment processes, and encouraging developers to write efficient code that minimizes computational resource usage.

In designing and implementing any process, we always factored in cost and sustainability, ensuring these processes aligned with these values.

Summary

When you search "continuous improvement" on Google, you'll come across various methods for implementing this in an organization, such as the Plan-Do-Check-Act (PDCA) cycle etc. However, in my view, there are two key elements essential for successfully implementing a continuous improvement cycle:

• Be able to identify, understand and learn from mistakes.
• Continuously strive for improvement in design and implementation, aiming to enhance them with each subsequent development or configuration iteration.

Our project, focused on greenfielding a tech organization, revolved around identifying, understanding, and learning from past mistakes to avoid repeating them. It was about retaining what was effective and discarding what wasn’t. System development is inherently complex, largely due to the rapid pace of technological advancement. Organizations often find it challenging to keep up with these evolving technologies. A key to achieving continuous improvement in your organization lies in building a robust framework for product development. This involves assembling the right team with an effective structure and ensuring they have access to ample operational data and information.