Adam VEGH
-
Get notification on Synapse Pipeline failures
I do love working in Synapse as I think this is a highly integrated orchestration tool that really contains almost everything that’s needed for a data engineer. However just as all tools it also has some missing pieces. One of those pieces is the alerting functionality at least a native build in alerting functionality. This…
-
Dynamic Schema Adjustment in Synapse Pipeline
In my previous blog posts () I’ve explained how to setup a dynamic pipeline with a control table. With that the ingestion job can be fully automated and scheduled for the appropriate frequency. In this post I’ll write a bit about how we can manage schema changes in the source system. Schema change is dependent…
-
Changing Fiscal Calendar with keeping the old one for on demand use
My company is currently changing it’s fiscal calendar. Due to this we have already received the requirement to adjust all of our reporting solutions. It wouldn’t be a huge problem at all, right? Just changing the calendar and map the days to different periods moving forward. The challenge has come when the requirement involved the…
-
Setting up a Dynamic Pipeline from Control Table
In the previous post I’ve explained how a Control Table should be setup up to handle necessary sources and processing behavior. In this post I’ll explain how to start creating a dynamic pipeline and add customized actions based on the table needs. When I’m setting up a dynamic pipeline I’m considering what actions should be…
-
I’m the data guy, but… which one exactly?
A lot of times I’m asked about my job by my relatives, friends or even a bank or administration office, and to be honest sometimes it is pretty hard for me to answer. Not that I wouldn’t know what I do, but when sometimes I’m saying I’m BI expert or Data Engineer they ask what?…
-
“Project Executives” in Power BI? Please, no!
Few years ago I experienced a methodology of implementing Power BI for the enterprise which I just couldn’t understand, why. Let’s call it “Project Executives”. It was a stream which’s purpose was to deliver Power BI reporting solution only for executives, and there were another stream – call it “Project Operations” – which was dedicated…
-
Create and Manage the Control Table for Dynamic Pipelines
As introduced in my other article I’m setting up dynamic pipelines to create sustainable and scalable flow for data integration. In this article I’m going to show you the first step for this, that is setting up the Control Table. The Control Table is the place where you can manage which tables should be collected…
-
Some Fun by practicing new formatting features and some custom visuals
This week I was practicing a bit in Power BI checking some new formatting functions and customer visuals. The topic was about Lego withing Maven Analytics Lego Challenge. I didn’t want to create an official one pager entry for the challenge, rather trying to put together a motion dashboard with some great features. Here are…
-
SQL Query is not enabled for On-Prem SQL Server
One of my current jobs required to pull the latest ID of a SQL table from an on-prem server. I thought it will be an easy one but finally it really wasn’t… let see why: To get the biggest ID from a SQL table is pretty easy with a SQL script, it is only like SELECT…
Adam's Perspective on Data 