A carregar...
Publicação
A Metadata-Based Framework for ETL Processes and Monitoring Implementation in the Insurance Sector
| Nome: | Descrição: | Tamanho: | Formato: | |
|---|---|---|---|---|
| 1.29 MB | Adobe PDF |
Orientador(es)
Resumo(s)
The Insurance sector is increasingly recognizing the value of data-driven decision-making in a
rapidly evolving digital landscape. This recognition has increased the demand for robust
Business Intelligence (BI) solutions to optimize operations and maintain a competitive edge.
In response, this thesis explores the implementation of a metadata-based framework for
Extract, Transform, Load (ETL) processes and monitoring within an Insurance Company,
utilizing Azure Cloud technologies. The primary goal is to create an end-to-end ETL process
that ensures efficient and accurate data flow, paired with a log registering and monitoring
system. The framework integrates Azure Data Factory for data extraction, Azure Databricks
for data transformation and Azure Data Lake Storage for efficient data storage. Central to the
solution is the Orchestrator, a custom-built tool using Azure Functions, which dynamically
manages and monitors ETL processes. The Power BI dashboard, designed for real-time
monitoring, provides detailed insights into the ETL pipeline, addressing key business questions
related to data flow efficiency, error rates and performance bottlenecks. This metadata-based
BI framework significantly enhances the operational efficiency of the insurance company. By
enabling real-time tracking and monitoring of data flows, the solution provides business users
with a clear and traceable view of data processes, enabling a data-centric culture within the
organization. Future work will focus on integrating advanced analytics and machine learning
capabilities to further optimize ETL processes, ensuring the framework's adaptability to
evolving business needs.
Descrição
Dissertation presented as the partial requirement for obtaining a Master's degree in Data Science and Advanced Analytics, specialization in Business Analytics
Palavras-chave
Business Intelligence Metadata Extract Transform Load Monitoring Kimball Lifecycle Insurance Sector SDG 9 - Industry, innovation and infrastructure