Our client, one of the largest companies in the German and European energy market, had a problem: Their status as industry leaders had caused their cloud infrastructure to become large and unwieldy. With thousands of people working remotely, onsite, and in offices across the globe, and with thousands of cloud resources to support it, the company’s virtual machines, microservices, databases, serverless applications, web applications, storage, and their maintenance, were driving wasteful time, making new digital initiatives and subsequent efficiency gains and streamlining more difficult to attain while driving cost.

At the same time, the increased complexity and growing demands of the energy sector required the organization to become more efficient and agile in the way they deployed and used their technology solutions. This gap between increasing cloud-computing requirements and stagnant cloud-computing capabilities led to networking problems, storage security risks, and governance issues. In this case study, the first out of two in the series, we will elaborate on how we solved automating cost optimization and cloud infrastructure security vulnerability remediation at scale through extensive use of low-code and no-code tools, DevOps and cloud-native storage services in Microsoft Azure cloud.

Key Obstacles and Challenges

One of the biggest challenges associated with large cloud infrastructure is dealing with large amounts of cloud resources in the IT environment and their compliance, governance, as well as approval processes at a large scale. This is evident in the energy sector, where the ability to securely store confidential data in a cost-effective, compliant manner is essential. Our client identified cloud security and cost-optimization as major concerns early in the process.

As the organization is handling and storing sensitive data, certain types of actions must first be approved by line managers, information security departments and other stakeholders before actually being implemented in the organization. At times, all staff did not know whether a certain action for closing a security loophole or introducing a cost-optimization action was possible and compliant with organization policies, forcing staff to reach out to managers, peers, teams and departments and other individuals they think may hold the answer by emails, calls, arranging meetings and sessions, many times more than one, to discuss the challenges and potential measures with managers, architects, IT security staff, case owners and other individuals who may, or may not, hold the answer, sometimes resulting in the answer that an action can be taken, is compliant and how it can be implemented, but many times that it cannot or that the action is not compliant with organization policies, leading to wasteful time. In regard to cost optimization, the costs in UAT and Dev environments were at times higher than expected, and staff often reverted to validating that certain types of expensive storage resource types were not created manually once every week to keep costs in check, a tedious process which could sometimes be described as a “continuous struggle”, which was further highlighted by the lack of a centralized overview of discovered issues, status and remediation progress.

Over the course of multiple years, as the organization grew and scaled up, multiple solution providers and consultants had been working on solutions across the four different environments, adding resources and configurations sometimes through automated CI/CD pipelines, but also often manually. Even though initiatives and policies were put in place mandating that all cloud infrastructure should be created through IaC templates, use private links etc., the processes and routines to enforce and monitor, and remedy existing resources, at scale were not there, leading to potential security vulnerabilities. Two distinct use cases for the project were thus identified: securing and hardening cloud infrastructure and cost optimization.

It was clear that the lack of automation, and with ever more cloud resources continuing to be added, was already driving significant wasteful time in the organization, and would risk was becoming an even bigger issue and blocker over time with an increasing amount of time spent to monitor the “green light” and status quo, and block new and other value-adding digital initiatives over time if nothing was done to achieve a higher degree of automation in the organization. Also, the more manual steps required, the higher the chance something may slip between the cracks, which could in turn lead to new and further security vulnerabilities. It was clear something needed to be done, which the management and senior IT managers at the client had realized.

But how can the cloud be used to optimize the cloud in terms of security and cost? How and in what way can cloud-native and open-source technologies be used to accommodate this type of process automation with success? How can an organization automate and standardize securing cloud infrastructure and custom solutions deployed over multiple years with little or no human interaction? The client decided to bring in one of our Axelsson Cloud Consulting experts to their project.

The Axelsson Solution

A key to any successful IT project is to properly map, refine and understand the actual needs of the organization, what is important, what actually brings value and understand potential pitfalls. In other, careful and thorough requirement engineering and refinement can go a long way in setting the foundation for a successful project. In this case, we also understood that understanding the process itself was important to make the right and informed technology choices on how to automate the processes and thus generate the expected result for the client. From an early stage, we identified a set of key success factors which we saw would be pivotal for the project’s success, namely, security, cost, structure and automation. Given the sensitive information processed by some of the resources, it was clear that we would need a high level of security, avoid public endpoints and ensure a deny-by-default approach to security in practice. As the organization had thousands of VMs, databases and other cloud resources deployed, we also quickly understood that the solution would need to be highly scalable, while keeping costs in check. The scale and amount of resources present in the IT environment also quickly made us understand that the ultimate value for the client would be had if all steps in the process could be automated and completed without human interaction by default.

Having worked closely with the client to map existing and new technical requirements, and having engineered and refined them into technical requirements, we came up with the distilled flow architecture below:

It was also clear that the different stakeholders, line managers, information security department in the case of a security vulnerability, and case owner, whom the organization had appointed the authority to enforce change when needed, would need to be able to review and approve/decline action with as great ease of use as possible for the users, and that multiple of them would need to perform an approval for a corrective measure to be implemented.

To ensure scale while maintaining cost, we opted for a serverless architecture as far as possible, using low-code Azure Logic Apps. Ensuring the highest possible degree of automation, the initial Logic App flow triggering the remediation is triggered by an Azure SQL table trigger for an event-driven, or a timer trigger allowing e.g., a manager to trigger a flow to identify and remedy issues on demand. The Logic App then retrieves, transforms and processes data from multiple sources including Log Analytics through Kusto (KQL) queries, internal HTTPS REST API calls, Azure SQL databases, Microsoft Entra ID (formerly Azure Active Directory) etc., for identifying and retrieving detailed information about resources with cost or security issues, determining the type of issue, contextualizing the issue with additional organizational data and retrieving the resource owner kid for the right work item user assignment respectively, as well as persisting the issues, their type and status in a master Azure SQL Server database and finally creating Azure DevOps work items assigned to the individual or team responsible for the security or cost issue.

Automating Approval and Remediation processes

The assignee can then choose to take a predefined remediation action, which will automatically apply the corrective measure to the affected resources through parameterized IaC templates, alternatively request a manual action, or request an exemption. In the case of an identified security vulnerability remediation, there could be multiple combinations of approvers needed, e.g., manager, case owner and information security department. For orchestrating the approval flow we once again used a Logic App, triggering standardized approval emails to the relevant approver(s) for the corrective measure. Upon approval, the Logic App captures any written comments by the approver, e.g., what action the assignee needs to take and how depending on issue and if requires manual action, or looks up a predefined comment depending on the type of identified issue, and writes it back to the DevOps work item as a comment, also making use of the default notification behaviour in Azure DevOps to notify the responsible team or individual, as any work item update triggers an email notification to the work item assignee in Azure DevOps by default. In the case of a predefined/suggested remediation action, the Logic App will finally invoke an Azure DevOps pipeline run where a parameterized Terraform IaC template will apply the corrective measure, upon which the Logic App updates the linked work item with a Closed status.

Observability and status tracking

For every action taken, the Azure SQL master database keeps a current record of the identified issues updates their pending issues and their progress. For certain types of corrective measures, especially cost optimizations in Dev, Test or UAT environments, actions could be taken without prior approvals, at which the corrective action can be taken immediately by the assignee. If an exemption is needed, and approved, the exemption is persisted in the master database for six months, where a look-up in the initial Logic App runs prevents work items from being created again for the type of issue that has been exempted, keeping the created work items requiring action relevant.

Ease of use

For selecting the aforementioned corrective measure, or manual action or exemption, we created a simple MVC app where the assignee can select the action template, request manual remediation action, or exemption, with a minimum 25-word justification description, triggering the Logic App approval flow when completed and available from within the work item through deep linking and inheriting the existing Azure DevOps authorization, in practice pre-authenticating the user for the application, ensuring as seamless user experience as possible when a manual step is required from the user.

Security

We adopted a no-compromise approach to security, ensuring that all resources created were virtual network and subnet integrated and using private links, which had also been designated by the organization. However, during the implementation phase, it became clear that due to the Logic Apps  tier used at the time, the consumption tier, we were potentially running into a security bottleneck, as Logic App Consumption tier does not support virtual network and subnet integration, in practice meaning that any user in the tenant by default can have access to the resource, even if configured with a managed identity, while if virtual network integrated, only principals within the same network can access the resource, which is more desirable from a security perspective. Given the already large amount of Logic Apps runs, and that each run costs a few cents each, we also saw a cost impact if staying in the existing tier. These issues were not unknown in the organization, however, the change had not yet taken place. To highlight the need for change, we created a presentation quantifying the impact cost-wise, given the current and future estimated amount of runs, as well as covering the security benefits of virtual network integration, something only available in the Standard tier. The change was introduced, with the total cost for the runs decreasing, while access security increased through network segmentation.

Throughout the project, working closely with the client, we also assisted in introducing new ways of working, including standardized naming conventions and their enforcement, as well as making sure all deployed infrastructure is created through IaC and that Azure Private Links are used for accessing Azure resources, further helping introduce a deny-by-default approach in practice.

Finally, we further extended the aforementioned master database with additional data, which was then used to create a PowerBI report, tailored for management and other stakeholders, enabling a full and visual overview of all identified, pending and closed security and cost issues in the organization.

Project Outcomes

Tedious and manual identification of cost and security issues have been replaced with a data- and event-driven approach. Sluggish approval and validation procedures have been replaced with quick and streamlined approval processes, sent to the right approver at the right time, depending on the type of issue. Limited or no high-level overview of identified issues and their status has been replaced with a fully visualized report of pending, ongoing and closed issues. Manual configuration of cloud resources and solutions to resolve identified cost or security issues has been replaced with automated remediation measures of cloud infrastructure through infrastructure-as-code.

The impact of business process automation and workflow transformation has been profound, with significant productivity and efficiency gains, as well as security improvements and thousands of Euros in cost savings thanks to better-sized and utilized cloud resources, with further cost savings accruing over time.

The automated workflows have allowed the client and its staff to spend less time on maintenance and keeping the light green and instead allowed the client to focus on more value-adding activities such as digital business models, driving further digital transformation throughout all parts of the organization and business units such as line operations, new digital innovation and automation for increased cost efficiency, further workflow improvements, predictive analytics, data-driven decision-making and more, enabling the client and its processes to be more agile, lean and streamlined, despite its large size, creating new and further competitive advantage.

The best part? This is only Phase 1. Look back here soon or sign up to our newsletter below, to find out how we adapted our automated solution to incorporate infrastructure creation and updates even in the most challenging scenarios in the Part 2 of this case study.

Ready to take the next steps?

Do you need a cloud expert for your project? Assessing how to gain opreational efficiences through digital technologies in your organization? Wanting to learn more on how we solved the use case in this case study for our client? Schedule a free 30-minute consultation to learn more about this project and discuss your options with us here

Business process automation is about eliminating wastes and bottlenecks, minimizing costs, reducing the chance of human error and achieving high efficiency and output, more with less, through systematized and streamlined processes, all pivotal to success in the energy market, other sectors, as well as for achieving climate and sustainability goals, such as net-zero. Looking to learn more? Subscribe to our newsletter to get our expert insights, as well as upcoming case studies, straight to your inbox.