Why does my Spark application slow down after running for a few days with a Zeppelin interface? - Stack Overflow

I have a Spark application running with an Apache Zeppelin interface. Initially, tasks execute quickly (in seconds), but after a couple of days of continuous operation, the same tasks start taking several minutes to complete. Restarting the Spark session temporarily resolves the issue, but the slowdown eventually reoccurs.

Here are some details about my setup:

• Spark Version: 3.2.3
• Zeppelin Version: 0.11.1

I’ve observed the following:

• There are no obvious errors or failures in the logs.
• Resource usage (CPU, memory, disk) doesn’t appear to be maxed out.

Questions:

1. What are the potential causes of this gradual slowdown in Spark performance over time?
2. Are there specific configurations or optimizations I should check to prevent this issue?
3. Could this be related to memory leaks, accumulation of metadata, or other long-running session issues?
4. Are there best practices for managing long-running Spark sessions with Zeppelin to avoid performance degradation?

Any insights, debugging tips, or recommendations would be greatly appreciated!