Issue with installing a package in updated hail environment

Hi,

I am using the newest version of the Hail jupyter notebook cloud environment. This comes with gnomad version 0.64. I need to update gnomad to the most recent version* and I need to install a few other packages, so I am using a startup script and pip. Unfortunately, this always fails and this is the message I get:

When I download the output script at the link, its always very long, but here is the relevant section with the error message:

ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/attrs-23.1.0.dist-info/METADATA'

I attempted to fix this by updating pip, and uninstalling and reinstalling attrs. This fixes the issue for attrs and most packages but there is always another package that has this problem.  For example:

+ docker exec jupyter-server /bin/bash -c 'pip install nbstripout             && python -m nbstripout --install --global             && git config --global core.excludesfile /home/jupyter/gitignore_global'
WARNING: The directory '/home/jupyter/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
Collecting nbstripout
  Downloading nbstripout-0.7.1-py2.py3-none-any.whl.metadata (18 kB)
Requirement already satisfied: nbformat in /opt/conda/lib/python3.10/site-packages (from nbstripout) (5.9.2)
Requirement already satisfied: fastjsonschema in /opt/conda/lib/python3.10/site-packages (from nbformat->nbstripout) (2.18.0)
Requirement already satisfied: jsonschema>=2.6 in /opt/conda/lib/python3.10/site-packages (from nbformat->nbstripout) (3.2.0)
Requirement already satisfied: jupyter-core in /opt/conda/lib/python3.10/site-packages (from nbformat->nbstripout) (5.3.1)
Requirement already satisfied: traitlets>=5.1 in /opt/conda/lib/python3.10/site-packages (from nbformat->nbstripout) (5.14.3)
Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.10/site-packages (from jsonschema>=2.6->nbformat->nbstripout) (23.2.0)
Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.10/site-packages (from jsonschema>=2.6->nbformat->nbstripout) (0.20.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.10/site-packages (from jsonschema>=2.6->nbformat->nbstripout) (70.0.0)
Requirement already satisfied: six>=1.11.0 in /opt/conda/lib/python3.10/site-packages (from jsonschema>=2.6->nbformat->nbstripout) (1.16.0)
Requirement already satisfied: platformdirs>=2.5 in /opt/conda/lib/python3.10/site-packages (from jupyter-core->nbformat->nbstripout) (3.10.0)
Downloading nbstripout-0.7.1-py2.py3-none-any.whl (15 kB)
WARNING: Error parsing requirements for pygments: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/Pygments-2.16.1.dist-info/METADATA'
WARNING: Error parsing requirements for asttokens: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/asttokens-2.2.1.dist-info/METADATA'
WARNING: Error parsing requirements for comm: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/comm-0.1.4.dist-info/METADATA'
WARNING: Error parsing requirements for exceptiongroup: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/exceptiongroup-1.1.3.dist-info/METADATA'
WARNING: Error parsing requirements for executing: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/executing-1.2.0.dist-info/METADATA'
WARNING: Error parsing requirements for importlib-metadata: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/importlib_metadata-6.8.0.dist-info/METADATA'
WARNING: Error parsing requirements for ipython: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/ipython-8.14.0.dist-info/METADATA'
WARNING: Error parsing requirements for jedi: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/jedi-0.19.0.dist-info/METADATA'
WARNING: Error parsing requirements for matplotlib-inline: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/matplotlib_inline-0.1.6.dist-info/METADATA'
WARNING: Error parsing requirements for parso: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/parso-0.8.3.dist-info/METADATA'
WARNING: Error parsing requirements for pexpect: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/pexpect-4.8.0.dist-info/METADATA'
WARNING: Error parsing requirements for prompt-toolkit: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/prompt_toolkit-3.0.39.dist-info/METADATA'
WARNING: Error parsing requirements for python-dateutil: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/python_dateutil-2.8.2.dist-info/METADATA'
WARNING: Error parsing requirements for pytz: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/pytz-2023.3.dist-info/METADATA'
WARNING: Error parsing requirements for stack-data: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/stack_data-0.6.2.dist-info/METADATA'
WARNING: Error parsing requirements for zipp: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/zipp-3.16.2.dist-info/METADATA'
Installing collected packages: nbstripout
  WARNING: The script nbstripout is installed in '/home/jupyter/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed nbstripout-0.7.1
+ docker exec -u 0 jupyter-server /bin/bash -c '/etc/jupyter/scripts/extension/install_jupyter_contrib_nbextensions.sh            && mkdir -p /home/jupyter/.jupyter/custom/            && cp /etc/jupyter/custom/google_sign_in.js /home/jupyter/.jupyter/custom/            && ls -la /etc/jupyter/custom/extension_entry_jupyter.js            && cp /etc/jupyter/custom/extension_entry_jupyter.js /home/jupyter/.jupyter/custom/custom.js            && cp /etc/jupyter/custom/safe-mode.js /home/jupyter/.jupyter/custom/            && cp /etc/jupyter/custom/edit-mode.js /home/jupyter/.jupyter/custom/            && mkdir -p /etc/jupyter/nbconfig'
Config option `kernel_spec_manager_class` not recognized by `EnableJupyterNbextensionsConfiguratorApp`.
Enabling: jupyter_nbextensions_configurator
- Writing config: /home/jupyter/.jupyter
    - Validating...
      jupyter_nbextensions_configurator 0.6.3 [32mOK[0m
Enabling notebook nbextension nbextensions_configurator/config_menu/main...
Enabling tree nbextension nbextensions_configurator/tree_tab/main...
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3061, in _dep_map
    return self.__dep_map
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2856, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3052, in _parsed_pkg_info
    return self._pkg_info
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2856, in __getattr__
    raise AttributeError(attr)
AttributeError: _pkg_info. Did you mean: 'egg_info'?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/jupyter-contrib", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/jupyter_core/application.py", line 285, in launch_instance
    return super().launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/traitlets/config/application.py", line 1073, in launch_instance
    app = cls.instance(**kwargs)
  File "/opt/conda/lib/python3.10/site-packages/traitlets/config/configurable.py", line 583, in instance
    inst = cls(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/jupyter_contrib_core/application.py", line 27, in __init__
    self._refresh_subcommands()
  File "/opt/conda/lib/python3.10/site-packages/jupyter_contrib_core/application.py", line 43, in _refresh_subcommands
    get_subcommands_dict = entrypoint.load()
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2473, in load
    self.require(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2497, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 792, in resolve
    new_requirements = dist.requires(req.extras)[::-1]
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2779, in requires
    dm = self._dep_map
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3063, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3072, in _compute_dependencies
    for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3054, in _parsed_pkg_info
    metadata = self.get_metadata(self.PKG_INFO)
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1475, in get_metadata
    value = self._get(path)
  File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1683, in _get
    with open(path, 'rb') as stream:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/Pygments-2.16.1.dist-info/METADATA'

Or

ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/platformdirs-3.10.0.dist-info/METADATA'

At this point, I'm reinstalling 25 packages in my startup script, which would be workable, but that last one, platformdirs, won't resolve even by deleting and uninstalling the package. I don't know why this would be. 

Additional details:

Terra account email address: archeney@ucsc.edu
Workspace Google Project ID: terra-037b9a34
Cluster ID:

Name: saturn-e0ba1503-1618-4575-a725-97f77bab5a5b
Workspace ID: 1de68429-527a-4c60-a614-836b141b502f

Screenshot of cloud environment configuration:


Link to workspace: https://app.terra.bio/#workspaces/terra-brca-exchange/CERFAC

 

*I need to update to the most recent version of gnomad, as 0.64 has a bug where the gnomad version 4 exomes dataset cannot be accessed. The purpose of my project is to work with gnomad version4 data, so this is unfortunately not a case where I can just use the older version- I am completely blocked here. 

(I did also attempt to update gnomad in the notebook instead, but that also fails when installing a dependency due to what I think is an unrelated issue.)

Thank you!

Comments

3 comments

  • Comment author
    Anthony DiCi

    Hi Allison,

    Thank you for writing in about this issue. Can you share the workspace where you are seeing this issue with Terra Support by clicking the Share button in your workspace? The Share option is in the three-dots menu at the top-right.

    1. Toggle the "Share with support" button to "Yes"
    2. Click Save

    Please provide us with a link to your workspace. We’ll be happy to take a closer look as soon as we can!

    Kind regards,

    Anthony

    0
  • Comment author
    Allison Cheney

    I have toggled it to "share"! Thank you!

    0
  • Comment author
    Allison Cheney

    Hi,

    Did I share it successfully? Or might there be an update on this? I would be grateful even to hear ideas for other things to try to get it to work.  Thanks!

    0

Please sign in to leave a comment.