• Kirill Smelkov's avatar
    *: Do not use relative imports · 3846997b
    Kirill Smelkov authored
    Because of the way wendelin.core organizes its in-tree python importing
    redirector (see wendelin.py) it is possible to import the same module
    twice with python thinking it is importing two different modules. For
    example when installed in develop mode python resolves the following
    imports to the same bigfile/__init__.py
    
        import wendelin.bigfile
        import bigfile
    
    but tries to load that module twice and independently. Which leads to
    virtmem DSO, linked to from under bigfile/_bigfile extension, being
    initialized twice and complaining about that because only single gil
    hook should be requested to be installed:
    
        (py39.venv) kirr@deca:~/src/wendelin/wendelin.core$ python
        Python 3.9.19+ (heads/3.9:40d77b93672, Apr 12 2024, 06:40:05)
        [GCC 12.2.0] on linux
        Type "help", "copyright", "credits" or "license" for more information.
        >>> import wendelin.bigfile
        >>> import bigfile
        python: bigfile/virtmem.c:106: virt_lock_hookgil: Assertion `!(virtmem_gilhooks)' failed.
        Аварийный останов
    
    This problem was there from day 1, but it was not creating issues in
    practice because wendelin.core users do `import wendelin...` and there
    was also no problem with running pytest in the source tree.
    
    However with py39 and pytest8 we see that running pytest somehow started
    to unconditionally import things from under two namespaces which leads
    to inability to run tests even when instructing pytest to collect them
    via python-modules namespace instead of filesystem:
    
        (py39.venv) kirr@deca:~/src/wendelin/wendelin.core$ pytest -vsx --pyargs wendelin.bigfile.tests.test_basic
        ======================== test session starts ========================
        platform linux -- Python 3.9.19+, pytest-8.2.2, pluggy-1.5.0 -- /home/kirr/src/wendelin/venv/py39.venv/bin/python3.9
        cachedir: .pytest_cache
        rootdir: /home/kirr/src/wendelin/wendelin.core
        configfile: pyproject.toml
        collecting ... python3.9: bigfile/virtmem.c:106: virt_lock_hookgil: Assertion `!(virtmem_gilhooks)' failed.
        Fatal Python error: Aborted
    
        Current thread 0x00007fa172a60740 (most recent call first):
          File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
          File "<frozen importlib._bootstrap_external>", line 1173 in create_module
          File "<frozen importlib._bootstrap>", line 565 in module_from_spec
          File "<frozen importlib._bootstrap>", line 666 in _load_unlocked
          File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
          File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
          File "/home/kirr/src/wendelin/wendelin.core/bigfile/__init__.py", line 31 in <module>
          File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
          File "<frozen importlib._bootstrap_external>", line 850 in exec_module
          File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
          File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
          File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
          File "<frozen importlib._bootstrap>", line 1030 in _gcd_import
          File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
          File "<frozen importlib._bootstrap>", line 972 in _find_and_load_unlocked
          File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
          File "<frozen importlib._bootstrap>", line 1030 in _gcd_import
          File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
          File "<frozen importlib._bootstrap>", line 972 in _find_and_load_unlocked
          File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
          File "<frozen importlib._bootstrap>", line 1030 in _gcd_import
          File "/home/kirr/local/py3.9/lib/python3.9/importlib/__init__.py", line 127 in import_module
          File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/pathlib.py", line 591 in import_path
          File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/python.py", line 492 in importtestmodule
          ...
          File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/runner.py", line 567 in collect_one_node
          File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py", line 837 in _collect_one_node
          File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py", line 974 in genitems
          File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py", line 811 in perform_collect
          File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py", line 349 in pytest_collection
          ...
          File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 178 in main
          File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 206 in console_main
          File "/home/kirr/src/wendelin/venv/py39.venv/bin/pytest", line 8 in <module>
        Аварийный останов (образ памяти сброшен на диск)
    
    This happens because wendelin.bigfile is importing
    wendelin.bigfile._bigfile as `from ._bigfile import ...` which under
    pytest leads to importing both wendelin.bigfile._bigfile and
    bigfile._bigfile and further conflicting when setting up GIL hooks.
    
    -> Fix this issue by avoiding relative imports and always referring to
       wendelin.core modules with `wendelin.` prefix.
    
    The list of places where relative imports were used was small and found via
    
        $ git grep -w import |grep '\s\.'
        bigfile/__init__.py:from ._bigfile import BigFile, WRITEOUT_STORE, WRITEOUT_MARKSTORED, ram_reclaim
        wcfs/__init__.py:from .client._wcfs import \
    
    Everywhere else we were already importing things from under wendelin
    namespace via fully specified module path.
    
    After the fix both
    
        $ pytest -vsx --pyargs wendelin.bigfile.tests.test_basic
    
    and
    
        $ pytest -vsx bigfile/tests/test_basic.py
    
    start to work ok from inside the worktree.
    
    /reported-and-tested-by @vnmabus
    /reviewed-by @levin.zimmermann
    /reviewed-on !26
    3846997b
__init__.py 24.9 KB