Test your AI knowledge

Hard-won SQL patterns for validating data, avoiding common anti-patterns, and thinking like a senior data engineer

Tuning and optimizing delta tables in a lakehouse can seem like a daunting task. Pipelines can start to slow down over time without reason. Unlike traditional relational databases, where you could set an index and forget it, Delta tables introduce new challenges. You are now dealing with an ever-growing number of parquet files stitched together by JSON files in your "_delta_log" folder and a few checkpoint files consolidating the changes every so often. The beauty of writing

Lessons from a career of cleaning up data dumpster fires. Imagine building a beautiful lake house on a pristine lake with gorgeous mountains in the surrounding distance, and a clean beach front in a relaxing neighborhood. Then in a couple of years, you find cracks in your foundation, the water becomes polluted by chemical runoff, and your property value plummets. This is exactly what happens when you build a data lakehouse poorly. All initial attempts start out well. Users ar