Dependency Management for Python Applications on Databricks
Databricks offers a powerful platform for distributed data processing. However, managing dependencies for Python jobs running on Databricks can be challengin...
Data Engineering
Use Dynamic Partition Overwrite for ETL with Apache Spark
After not using Apache Spark at all in 2019, I am currently catching up on features and improvements I missed since version 2.1. While pandas UDFs are certai...
Speedup R by moving temporary files into memory
There are certain situations when you need to work with temporary files in R. For instance, my package Jaatha requires that an external simulation tool is ca...