Apache Spark Cluster Mode Error: NoSuchFieldError: METASTOREURIS

I am running Apache Spark with Hive Metastore on YARN. The setup consists of:

• One Edge Node (running inside containers, using --network host mode):
  • Hive Metastore
  • PostgreSQL (as the Hive Metastore database)
  • Jupyter Notebook
  • YARN ResourceManager in host
• Five Worker Nodes running HDFS and YARN NodeManager.

Problem

When running a Spark job in client mode, everything works fine.
However, when switching to cluster mode, the job fails with the following error:

Traceback (most recent call last):
  File "/usr/local/hadoop/yarn/nodemanager/data/usercache/spark/appcache/application_1738582462412_0011/container_1738582462412_0011_02_000001/test_query.py", line 8, in <module>
    spark.sql("SHOW DATABASES").show()
java.lang.NoSuchFieldError: METASTOREURIS

Hive Metastore Configuration in Spark (spark-defaults.conf)

spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.spark_catalog.type=hive
spark.sql.catalog.spark_catalog.uri=thrift://hive-metastore:9083

Networking Container Details

Since I am running Hive Metastore in a container with --network host, I added the hostname mappings manually:

127.0.0.1       localhost
--add-host hive-metastore:192.168.225.208
--add-host spark-iceberg-notebook:192.168.225.208
--add-host dw2endev01.sme.astro:192.168.225.208
--add-host dw2endev01:192.168.225.208
--add-host dw2nndev01.sme.astro:192.168.225.209
...

Networking Host Details

127.0.0.1 localhost
127.0.0.1 hive-metastore
127.0.0.1 spark-iceberg-notebook

192.168.225.208 dw2endev01.sme.astro dw2endev01 spark-iceberg-notebook hive-metastore
192.168.225.209 dw2nndev01.sme.astro dw2nndev01 sme-hadoop-cluster
192.168.225.210 dw2nndev02.sme.astro dw2nndev02
...

Troubleshooting Done

• The same Spark job runs successfully in client mode, but fails in cluster mode.
• Checked /etc/hosts resolution on worker nodes, and hive-metastore resolves correctly.

Question

What could be causing the java.lang.NoSuchFieldError: METASTOREURIS error in cluster mode?
Is it a classpath conflict or a networking issue between Spark and Hive Metastore?