Spark Config Madness

Will it Ever Stop?

I’ve been on a spark config kick lately. A week or so ago, I published a post showing how to config Spark for the AWS Glue Iceberg Rest endpoint. Overall, the config is not too complex, but does require a ton of inputs. Comparing that to the duck, you see a stark contrast.

However, after working through that article, I was somewhat disappointed that the …