Test your AI knowledge

Lessons from a career of cleaning up data dumpster fires. Imagine building a beautiful lake house on a pristine lake with gorgeous mountains in the surrounding distance, and a clean beach front in a relaxing neighborhood. Then in a couple of years, you find cracks in your foundation, the water becomes polluted by chemical runoff, and your property value plummets. This is exactly what happens when you build a data lakehouse poorly. All initial attempts start out well. Users ar

Six patterns that make your Spark pipelines more reliable, portable, and easier to maintain I have worked as a Data Engineer for several years and have made my share of mistakes along the way. However, with experience, I have learned a few key tricks that really help you build reliable, clean, and reproducible code. This article focuses on a few powerful concepts that you should master as you build a career as a data engineer. 1. Know Your Platform Utilities: dbutils vs. no