python - pytest throwing module no found in Databricks' notebook - Stack Overflow

I'm recreating below test scenario but constantly getting below error:

ImportError while importing test module '/Workspace/python_tests/dummy_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'dummy_test'
=========================== short test summary info ============================
ERROR dummy_test.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.45s ===============================

This is the structure of my /Workspace/python_tests directory and each file. All 3 files reside in the same directory:

1. dummy.py

import pyspark
from pyspark.sql import SparkSession
from pyspark.sql.functions import col

# Because this file is not a Databricks notebook, you
# must create a Spark session. Databricks notebooks
# create a Spark session for you by default.
spark = SparkSession.builder.appName('integrity-tests').getOrCreate()

def dummy():
    return "I am a dummy"

2. dummy_test.py

# import pytest
import pyspark
from dummy import *


def test_dummy():
    assert dummy() == "I am a dummy"

3. pyTestRunner (notebook)

%pip install pytest

import pytest
import sys

# Skip writing pyc files on a readonly filesystem.
sys.dont_write_bytecode = True

# Run pytest.
retcode = pytest.main([".", "-v", "-p", "no:cacheprovider"])

# Fail the cell execution if there are any test failures.
assert retcode == 0, "The pytest invocation failed. See the log for details."

I'm recreating below test scenario but constantly getting below error:

https://learn.microsoft.com/en-us/azure/databricks/notebooks/testing

ImportError while importing test module '/Workspace/python_tests/dummy_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'dummy_test'
=========================== short test summary info ============================
ERROR dummy_test.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.45s ===============================

This is the structure of my /Workspace/python_tests directory and each file. All 3 files reside in the same directory:

1. dummy.py

import pyspark
from pyspark.sql import SparkSession
from pyspark.sql.functions import col

# Because this file is not a Databricks notebook, you
# must create a Spark session. Databricks notebooks
# create a Spark session for you by default.
spark = SparkSession.builder.appName('integrity-tests').getOrCreate()

def dummy():
    return "I am a dummy"

2. dummy_test.py

# import pytest
import pyspark
from dummy import *


def test_dummy():
    assert dummy() == "I am a dummy"

3. pyTestRunner (notebook)

%pip install pytest

import pytest
import sys

# Skip writing pyc files on a readonly filesystem.
sys.dont_write_bytecode = True

# Run pytest.
retcode = pytest.main([".", "-v", "-p", "no:cacheprovider"])

# Fail the cell execution if there are any test failures.
assert retcode == 0, "The pytest invocation failed. See the log for details."

• python
• pytest
• databricks

Share Improve this question Follow edited Jan 12 at 22:14 marcin2x4 asked Jan 11 at 14:48 marcin2x4marcin2x4 1,45933 gold badges2929 silver badges5656 bronze badges 5

• Error will probably only occur when tests are residing in a sub folder. Pytest doesn't work as expected regarding relative imports. This can be worked around executing pytest module as a script, like python -m pytest python_tests_folder. In a Databricks context this may be different. More information about the -m option can be found here: stackoverflow.com/questions/7610001/… – Bouke Commented Jan 12 at 19:58
• Didn't help, same error appears. – marcin2x4 Commented Jan 12 at 22:21
• 1 The article mentions that this needs to be done in a repo. Can you confirm you are creating these files and running the tests in a repo on Databricks? – Anupam Chand Commented Jan 14 at 7:45
• Yes, this is the structure of my dbr's worksapce. – marcin2x4 Commented Jan 14 at 21:29
• You need to build this structure within a repo. It should work then. – Anupam Chand Commented Jan 15 at 21:59

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

I had the same issue when I found this post. I was able to resolve it by moving dummy_test.py into a subfolder of the tests folder from which I execute pytest (i.e., a subfolder of the directory where your pyTestRunner notebook lies). So:

tests/
    py_test_runner.ipynb
    dummy_tests/
        dummy_test.py

Hope this helps someone encountering the same issue and landing at this question.