Ace your next Spark interview with six whiteboard sketches
You cleared the recruiter screen. The hiring manager liked you. Then the technical round starts, and someone asks you to explain the difference between coalesce and repartition.
You know the answer. You’ve used both a hundred times. But somehow, it comes out tangled, and you can watch the interviewer lose interest in real time.
We’ve all been there. Knowing something and explaining it clearly under pressure are two different skills.
As a data engineer in Chicago with five years of Spark experience, I’ve seen the same handful of questions come up again and again.
The fix is to have a simple mental picture for each core concept. You need one you can sketch on a whiteboard in 30 seconds. This article covers six of the most common Spark interview questions. Each one comes with a short answer and a diagram (with a little Chicago flavor) that you can draw during the interview. Study them before your next interview, and you’ll walk in with answers that are easy to learn and repeat.
1. What is the difference between a transformation and an action?
Spark doesn’t run your code line by line. When you call select, filter, join, or withColumn, nothing actually happens. Spark simply adds that step to a logical plan. These are transformations, and they’re lazy.
The work starts only when you call an action such as count(), show(), collect(), or write. Waiting until an action to do any work is called lazy evaluation.
Why does this matter? You will eventually have to run the job, or you won’t be employed very long. The payoff is in the planning. Because Spark sees the whole chain of steps before running anything, the Catalyst Optimizer can turn it into an efficient blueprint called a DAG (directed acyclic graph). It can push filters down to the data source, read only the columns you need, and reorder steps.
You can help it along. Filter out unneeded rows and select only the columns you need as early as possible. The smaller your data is before Spark reaches an action or an expensive step like a join, the faster the job runs.
Interview answer: Transformations like select and filter are lazy and only build a plan. Actions like count() and write trigger execution. Because Spark waits, the Catalyst Optimizer can optimize the whole DAG first, and filtering rows and pruning columns early makes the job even faster.
2. What are narrow and wide transformations?
This question is really about how data moves.
In a narrow transformation, each output partition comes from exactly one input partition. Spark processes every partition right where it sits, so the data stays on the same machine. Commands like map, filter, and union are narrow. Think of them as separate L lines running in parallel. In the example, each train stays on its own track.
In a wide transformation, each output partition needs data from many input partitions. Spark has to move data across the network between executors, which is called a shuffle. Commands like groupBy, join, and distinct are wide. Think of three pizzerias delivering to every neighborhood in the city. Every pizzeria has orders from every neighborhood, so delivery routes crisscross the city. Unlike a narrow transformation, the data can’t stay in its own lane.
Why does it matter? As usual, it all comes down to performance. Spark chains narrow transformations together and runs them in a single pass. Every wide transformation forces a shuffle, which means writing data to disk, sending it over the network, and starting a new stage. Shuffles are usually the slowest and most expensive part of a job, so fewer shuffles mean faster, cheaper queries.
Interview answer: Narrow transformations work within a single partition, so there is no data movement. Wide transformations need data from multiple partitions, which causes a shuffle. Moving data across the network is expensive and hurts performance.
3. What is the difference between repartition() and coalesce()?
Both change the number of partitions, but they work very differently.
The coalesce() command is the lightweight option. It can only reduce the number of partitions, and it does this by merging partitions that already sit on the same executor, without a full shuffle. Picture streams joining into a river. Some rivers end up bigger than others, and so do the merged partitions.
The repartition() command is the heavy-duty option. It performs a full shuffle, so it can increase or decrease the number of partitions, and it spreads the data evenly. You can even choose a column to partition by, such as repartition(200, "customer_id"). Picture a man-made lock that collects all the water and releases it evenly into new channels.
As a simple rule, use coalesce to cut down the number of output files before a write. Use repartition when you need more parallelism or have to balance lopsided partitions.
Interview answer: Coalesce only reduces partitions by merging neighbors without a shuffle. It’s fast but can be uneven. Repartition does a full shuffle, which is more expensive but creates much more even partitions.
4. How does a broadcast join work, and when should you use one?
As a rule of thumb, use a broadcast join when you’re joining a large table to a small one.
In a normal sort-merge join, Spark shuffles and sorts both tables so that matching keys land on the same executor. That’s expensive. In a broadcast join, Spark sends a full copy of the small table to every executor. Each executor then joins its own slice of the large table locally, and the large table never moves.
Spark does this automatically when the small table is under the spark.sql.autoBroadcastJoinThreshold setting, which is 10 MB by default. You can also request it directly in code:
from pyspark.sql.functions import broadcast
orders.join(broadcast(stores), "store_id")
However, for this to work properly, the small table has to fit in memory on every executor. That is why you normally broadcast small dimension tables as opposed to large fact tables.
For senior-level depth, note that Spark can skip the sort-merge shuffle entirely if both tables were saved with bucketBy on the join key and the same number of buckets.
Interview answer: Spark copies the small table to every executor, so each one joins its slice of the large table locally and nothing large gets shuffled. I use it when one side is small enough to fit into memory.
5. What is data skew, and how do you fix it?
Data skew happens when a few partitions hold far more data than the rest. The tasks working on those partitions run much longer, and the whole job waits for them to finish.
Think about Illinois. Chicago has about 2.7 million residents, roughly 2.5 million more than Aurora, the state’s second-largest city. If every city had just one post office, Chicago’s would be overwhelmed while the others sat nearly idle. Partitioning data on a lopsided key creates the same problem.
That’s why choosing a good partition key matters. Dates are a popular choice because they often spread data evenly, but they can backfire if your business is seasonal. A retailer’s December partition might dwarf its July partition.
A few ways to fix skew:
• Use Adaptive Query Execution (AQE). With spark.sql.adaptive.skewJoin.enabled turned on, Spark splits oversized partitions in joins automatically.
• Salt the hot key. Add a random number from 0 to N to the key on the large table, copy the small table once for each number, and then join on the key plus the salt.
• Broadcast the small table. If the other side of the join is small, a broadcast join avoids the skewed shuffle entirely.
• Handle problem keys separately. Nulls and default values are common culprits, so filter them out and process them on their own.
AQE fixes skew at runtime by checking real partition sizes after each stage, while salting is a manual fix written into your code. Try AQE first, but it only covers joins, so skewed groupBy aggregations still need salting.
Here’s what salting looks like in PySpark, splitting the hot Chicago key into four pieces:
from pyspark.sql import functions as F
N = 4 # number of salt values
# Large table: add a random salt (0 to 3) to every row
trips_salted = trips.withColumn("salt", (F.rand() * N).cast("int"))
# Small table: make one copy for each salt value
salts = spark.range(N).select(F.col("id").cast("int").alias("salt"))
cities_salted = cities.crossJoin(salts)
# Join on the original key plus the salt
joined = trips_salted.join(cities_salted, ["city", "salt"])
Chicago’s rows now spread across four partitions instead of one, and each one still finds its match because the small table has a copy for every salt value.
Interview answer: Skew is when a few partitions hold most of the data, causing a few tasks to run far longer than the rest. I fix it with AQE skew joins, salting hot keys, broadcasting small tables, or handling nulls separately.
6. What is the difference between cache() and persist()?
This one trips people up because the answer is that they do the same thing, with one small difference.
Because of lazy evaluation, Spark doesn’t keep a DataFrame around after using it. Every action recomputes it from the source. If you build three reports from the same DataFrame, Spark reads and processes the data three times. Caching saves the result after the first computation so later actions can reuse it. It’s like a restaurant prepping ingredients before the dinner rush instead of starting from scratch for every order.
Here is the difference:
• cache() uses the default storage level. For DataFrames, that means memory first, with any overflow spilling to disk.
• persist() lets you choose where the data goes: memory only, disk only, both, or replicated across executors.
In other words, cache() is just persist() with the default setting.
trips.cache()
trips.count() # first action stores the data
trips.groupBy("station").count().show() # reuses the cache
trips.unpersist() # free the memory when done
Three things to remember:
• Caching is lazy, too. The data is stored only when the next action runs.
• Cache only what you reuse. Caching a DataFrame you use once just wastes memory.
• Clean up. Call unpersist() when you’re done so the memory goes back to your other jobs.
Interview answer: Both store a DataFrame so later actions reuse it instead of recomputing it. The cache() command uses the default storage level, while persist() lets me choose memory, disk, or both.
Final thoughts
Interviewing well comes down to preparation. Memorizing definitions might get you through a few questions, but it falls apart the moment an interviewer asks “Why?” or “When would you use that?”
Understanding the fundamentals holds up. If you can explain lazy evaluation, shuffles, partitioning, joins, skew, and caching in plain language, you’re almost there. If you can sketch a quick picture of each, you’ll really get the interviewer’s attention.
Finally, make sure to keep your answers short. A clear two-sentence answer and a quick sketch beat a five-minute lecture every time. If the interviewer wants more detail, they’ll ask, and now you’re having a conversation instead of taking a quiz.
Learn the picture, and the words will follow.
Do you have a Spark interview tip that helped you land a job? Share it in the comments. As developers, we’re all lifelong students.
Comments