Writing

An end-to-end, copy-pasteable runbook for setting up Databricks file events on AWS using your own SQS queue (Provided queue mode), direct S3 → SQS notifications, Unity Catalog external locations, file arrival triggers, and Auto Loader.

How Snowflake can read Delta tables governed by Unity Catalog using Iceberg Uniform reads — via credential vending, external volumes, and object storage catalog integration.

In Databricks, metadata of data Objects (tables, views, etc.) is registered in Metastore. Previously, Databricks uses Hive metastore by default to register schemas, tables, and views. However, it's highly recommended to upgrade…

Delta Lake is an open source storage layer that brings reliability to data lakes. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. Delta Lake runs…

Most Big Data developers and Data Engineers start learning Spark by writing SparkSQL codes to perform ETL on DataFrame (I know I did). I also wrote a post about SparkSQL Programming. However,…

TL,DR SparkSQL is a huge component of Spark Programming. This post introduces programming in SparkSQL through Spark DataFrame API. It's important to be aware of Spark SQL builtin functions to be a…

You Spark job is running for a long time, what to do? Generally, longrunning Spark jobs can be due to various factors. We like to call them the 5S Spill, Skew, Shuffle,…

This tutorial follows Debezium official documentation Deploying Debezium on Kubernetes, but modified for Azure Kubernetes Service and Azure Container Registry. You need an Azure account, Azure subscription, and Azure resource group before…