 is the use of technology to radically improve the performance or reach of an enterprise. One of the first statistical analysis about Digital Transformation of 391 different companies with different level of Digital success concluded that there are 2 principal dimensions that measure the success of Digital Transformation — the what of technology (Digital Capability) and the how of leading change (Leadership Capability).

 have not adopted any digital technologies, nor have they properly led that change.

 have aggressively adopted the technologies, but lack the leadership to give that technology any business context.

 try to lead a digital change without introducing new digital technologies.

 excel in both dimensions. Digital Masters are 26% (EBIT margin) more profitable than their industry peers thanks to not only adopting new technology but also having Leadership Capability to manifest the change.

There are 4 distinct components to Digital Capability.

 can be enhanced via digital technology by creating and communicating brand vision of the company, identifying customer touch points and creating cohesive personalized experience through many digital channels — social media, web, mobile, e-commerce. Additionally, digital feedback can be used for customer-driven innovation.

 can be digitally transformed by automating business core processes, changing the way employees work, creating real-time transparency, and making informed decisions.

 — what you sell, how you sell and how you make money from it — can be reinvented by digital changes. Physical products and services can be replaced with digital equivalents (e-services, e-commerce). Industries can be reinvented using shared economy concept (Airbnb, Uber, Bolt), enabled by digital technology.

 is a prerequisite implementation to enable all the beforementioned components.

Specifically, there are 5 distinct domains of digital transformation that, by using effective Digital Strategy and corresponding Digital Capability, can be used to digitally transform the business.

 are not any more static set of consumers, but have a two-sided relationship with the business. An effective 

 is to identify all the customer touchpoints and make use of technology to create more value. Access to your product or service can be improved by having omni-channel experience (web, mobile, cloud, in-demand service). Customers can be engaged by thinking like a media company and earning attention of your audience. Personalized experienced can be provided by recommendation systems and personalized products, content and offerings. Likewise, customers give an active feedback to the business by writing reviews, engaging with the product and producing behavioral data for analytics.

 is highly related to cooperation in Digital Age due to the rise of Platform Business Model (Airbnb, Bolt, Uber). The 

 is either to build a store or a marketplace. Due to the scalable nature of technology (cloud, big data, analytics) it is possible to provide value on a multi-sided platform by leveraging the platform for your business case. The platform usually provides simplicity, standardization, cooperation for the rivals to “grow the pie” while the rivals compete with each others to “divide the pie”.

 has become the king. The rapid development of new capabilities for managing and making sense of this has been shaped by the rise of cloud computing, which makes the potential of big data accessible. The 

 is to put data in the cloud in service of innovation and value creation by warehousing, parallel processing and training models for machine learning and natural language processing purposes. At the same time data strategy must take into account the legal, risk management, and data security.

 is the process by which new ideas are developed, tested and brought to the market by businesses. Digital Capabilities enable rapid experimentation. The 

 makes use of both — exploratory learning (divergent experiments) and confirmatory learning (convergent experiments). Divergent experiments are focusing on minimum viable product and iterating continuously — being open minded with minimal cost and maximum learning. Convergent experiments are used for hypotheses testing using A/B testing, multivariate analyses on analytics data between test and control group.

 of the business is threatened by emerging technology. As digital technology is empowering customers’ options, it’s important for the business to react accordingly. The 

 is to re-iterate the value proposition of the business in the light of new emerging technologies and adapt. It means to proactively identify key customer types, existing value provided, emerging threats, potential new value generated and generation of new value proposition that has hopefully turned the threat into opportunity.

Similar to Digital Capability there are 4 distinct components to Leadership Capability.

 is a seed that is planted at the top of the company engaging people at all levels to make the vision live and grow. It is the cornerstone of a successful Digital Transformation. Digital Masters have transformative digital vision. They are aware of current threats and opportunities in their industry. They share the vision, make it focused on business, not technology and are aiming this to be transformative while specific.

 is needed to create a source of inspiration and crowdsource people to co-create solutions using collective intelligence. Leading by example, setting expectations and recruiting believers to join in. Focus should be on the critical conversations to solve business problems, not on tools themselves. Practical outcomes are trainings, hackathons, content moderation, scorecards with key performance indicators, a digital roadmap.

 is needed to iteratively coordinate and share the initiatives across the company. It means prioritizing and aligning practical initiatives and common resources across the company. Someone must be placed in charge of leading a Digital Transformation (e.g. CDO, CTO). A suitable governance mechanism is decided — committees, leadership roles, business units, shared digital units.

 helps to translate the digital vision into a vision of a technology platform as the foundation of Digital Transformation. Clean, well-structured digital platform contains data, data definitions, systems, DevOps practices, etc. Everything starts from building out the right digital skills to meet the requirements of digital economy, establishing a shared technical understanding, IT-business relationship, possible partnerships.

One of the biggest obstacles to Digital Transformation is the failure to recognize the need for change, the urgency. Businesses react to threats instead of shaping the future. It is understandable because it’s hard to orient in the middle of continuous technological hype.

On a high level Digital Transformation comes down to 3 following steps:

When the starting point is clear — your position on the Digital Transformation Matrix — it’s time to create vision to become Digital Master. This vision must be shared and translated to roadmap of actionable landmarks while taking into account existing assets and capabilities. While engaging, explaining, aligning and sending signals it’s important to stop telling and start showing by finding quick wins to prove the concept. Finally it’s important to encourage informed risk-taking, learning to fail.

The structure of Digital Transformation

Digital Transformation

One of our client companies streams a high volume of real-time business-critical events using the Kafka ecosystem. For security reasons, the company is using a “collector application” that uses a JSON Schema document such that it is only possible to publish events to topics that are whitelisted in that document.

Currently the business partners are not getting access to the data soon enough to make decisions due to the high latency (from hours up to days) as the data is batch-processed using Amazon S3 and Apache Spark on Amazon EMR cluster. Additionally, the inability of the current system to handle traffic at this scale results in significant data loss, in some cases as high as 15%. From a security perspective, all the topics are available to all partners right now, whereas they should only be accessible to the partners who actually own the topic. Finally, the solution is not very adaptable to changes right now. Introducing new topics/events to the system is a manual process, which is time-costly and prone to error.

Thus we proposed and proved at scale a new metadata-driven architecture with the objectives to improve the latency, scalability, cost, security, data integrity and adaptability of the system.

The intended audience of the article is Data Engineers, Data Architects and Solution Architects.

The proposed solution is to build out and automate a Data Lakehouse using metadata-driven architecture without changing the existing JSON Schema document. This solution consists of multiple concepts that will be explained next.

A Data Lake is a data storage repository that can store large amount of structured and semi-structured data. The Data Lake usually contains many layers and is implemented using the native storage of a cloud provider. For example, Amazon S3 can be used to build a data lake using AWS cloud.

The Data Lake ingests the data from multiple sources and stores the data unmodified in ingestion layer (also called as raw layer or bronze layer). The Data Lake has some intermediate layers to perform data transformations. The further the data is moved in terms of layers, the closer the data is getting to its eventual use case, whether it’s building a BI report, an ML model or the predictions of that ML model. There is no hard limit to the number of layers in that architecture.

A Data Lakehouse combines the Data Lake with the Data Warehouse into a single cohesive data platform. Due to its unique set of features, 

 is one of the leading Lakehouse platform. Given an AWS as the cloud platform, Snowflake can leverage Amazon S3 as a storage resource. It is possible to create a 

 in Snowflake and reference to it as a 

. Thus all the permanent objects are eventually stored in Snowflake managed S3 buckets. The data lake layers can be created in Snowflake as separated databases

As a solution, we created a code generation engine that takes the existing JSON Schema as input, generates the DDL and executes the statements by the engine producing an infrastructure for all the defined entities in the configuration document. The solution has 2 sources — an AWS MSK and a JSON Schema.

The AWS MSK is a Kafka cluster where all of the company’s real-time event-streaming is done. The JSON Schema is the company’s existing object that whitelists all the allowed topics in the Kafka Cluster together with additional attributes, defining ~300 topics. In addition to the name of the topic, each configuration specifies the owner of that topic and set of roles who can see all the fields in the event in an unmasked manner. As this is metadata/configuration driven architecture, the JSON Schema defines what is happening in other parts of the system.

The Configuration table stores currently active versions for all of the topics. As per the JSON Schema, there are going to be 300 active configurations/rows in that table. The system is using Snowflake 

 to capture the change in configuration tables. When changes are encountered, a Snowflake 

 is executed to build out all Snowflake components for that specific topic.

The Raw message table serves as the raw layer of the Data Lakehouse consisting of a VARIANT column supporting semi-structured data and is the landing zone for the events coming from Kafka cluster for one specific topic. The system creates a 

 from the JSON Schema, and applies it to the raw message table. From security perspective, this means that this table as well as the objects derived from that table have specific fields masked (e.g. PII fields). Only dedicated roles specified in the JSON Schema can see unmasked data.

The system uses a Kafka Connect Cluster to ingest the data from a topic to the raw message table. We created a containerized Kafka Connect instance that is using the 

 implementation for loading the data. The Kafka Connect Cluster forms out of many instances and can thus be horizontally scaled in an automatic manner based on the load. The running cluster exposes a REST API that can be used to configure the Kafka Snowflake Connector inside the cluster.

As this architecture must be metadata driven, the system must be able to respond to the changes in the metadata. We created a Utility Service to perform this function. For example, when new topic is added to the JSON Schema, the Utility Service updates Kafka Connect Cluster via the REST API to also loads the data from the newly added topic into Snowflake.

To be successful, the system had to have an average latency between the Kafka event and the Snowflake table of no greater than 5 minutes, so that critical business decisions could be made in a timely manner. We managed to over-deliver on this goal by implementing a near real-time solution using micro-batching instead of earlier batch-processing.

The combination of Kafka Connect Cluster’s horizontal scalability and Snowflake’s serverless 

 feature enabled us to benchmark the system during performance tests on 41 topics with ~4 million events per minute reaching Kafka. Cost estimations under both heavy and normal loads show a significant reduction compared to the previous solution. That way we configured the solution to handle the peak load and be scalable. Given the performance and cost estimations we managed to predict the yearly cost of that solution for the company.

One of the biggest game changer is the new security policy we introduced. Using the Dynamic Masking Policy, the sensitive data is now masked by default in the first layer in data lake, and all subsequent layers. Additionally, the data was only accessible to the owner of the specific topics. Being masked, it is not even accessible to Administrators.

Compared to the previous solution, which suffered from a data loss of up to 15%, during the performance test of the new system with a total load of ~230 million events, no data loss was detected.

Finally, the solution is now metadata-driven such that implementing a change in the metadata causes the system to adapt to the change without any manual intervention.

The metadata-driven architecture to build out and automate Data Lakehouse turned out to be an excellent and performant solution to adapt the changes and fulfill the Company’s requirements in terms of latency, scalability, cost, security, data loss and adaptability of the system.

We believe that implementing similar metadata-driven architecture in systems where new events are likely to be introduced in the future is less error-prone, decreases the development effort and guarantees fast availability of data in the Data Lake.

How we used metadata-driven architecture to build out and automate Data Lakehouse

Metadata-driven architecture for Data Lakehouse

Building Digital platform for the city of Wuppertal, Germany. The digital platform contains public mobile application for the citizens of Wuppertal and back-office solution for managing the content and providing different features of the digital platform.
Full-stack distributed development - teams in Estonia, Lithuania, Germany. Load testing implemented using Gatling.

Bliggit Wuppertal

Implementing data acquisition logic for eCommerce data to bring the data to the Data Warehouse layer for further analytics.
Participating in the Full Stack Development of a system to replace an existing batch processed solution with streaming solution using Kafka. The solution is used to calculate commissions for frontline workers based on their performance and other set of criteria
Building a big data solution to handle peak time volume of 3 million events per minute and thus increase the performance of the previous system by implementing stream processing capabilities. Participating in the Development of a system to build out and automate Data Lakehouse. Implementing supportive services and benchmarking the performance of the solution.

Metadata-driven Data Lakehouse

Built a comprehensive ML platform for predictive analytics and model deployment. The platform includes feature extraction, automated model training and testing, model versioning, and deployment pipelines. Integrated LLM models for specific use cases and built AI agents for business automation.

Customer Growth Platform

Building a business onboarding and compliance platform. Implemented integrations with compliance providers to provide KYC, ongoing monitoring, identity document verification process. Enabled data analytics by building change data capture (CDC) and data streaming into Snowflake data warehouse.

Binderr

To put data into use in terms of business objectives we leverage modern machine learning and AI methods.

Machine Learning & AI

Data is the new gold and is the most important input for high-quality output solutions - be it analytical reports or machine learning models.

Data Engineering & Analytics

To build reliable, scalable solutions, we provide the tried-and-tested foundation and operations using cloud-native technologies.

DevOps & Cloud

Bringing your ideas to life, being it a new product, internal workflow modernization or third-party integration.

Custom Development

We deliver user-centric mobile experiences optimized for cross-platform environments. Our agile development approach ensures adaptability to evolving requirements.


Mobile Development

Good UX/UI design is an outcome of constant refinement. We have an agile process to rapidly test out ideas and iterate.

Aspensource

Building Digital Products
& Transformations

Using skilled talent, we provide you personalized experience, innovation and digital vision

Building Digital Products & Transformations

Using skilled talent, we provide you personalized experience, innovation and digital vision

Building Digital Products
& Transformations