Issue with installing a package in updated hail environment
Hi,
I am using the newest version of the Hail jupyter notebook cloud environment. This comes with gnomad version 0.64. I need to update gnomad to the most recent version* and I need to install a few other packages, so I am using a startup script and pip. Unfortunately, this always fails and this is the message I get:
When I download the output script at the link, its always very long, but here is the relevant section with the error message:
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/attrs-23.1.0.dist-info/METADATA'
I attempted to fix this by updating pip, and uninstalling and reinstalling attrs. This fixes the issue for attrs and most packages but there is always another package that has this problem. For example:
+ docker exec jupyter-server /bin/bash -c 'pip install nbstripout && python -m nbstripout --install --global && git config --global core.excludesfile /home/jupyter/gitignore_global'
WARNING: The directory '/home/jupyter/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
Collecting nbstripout
Downloading nbstripout-0.7.1-py2.py3-none-any.whl.metadata (18 kB)
Requirement already satisfied: nbformat in /opt/conda/lib/python3.10/site-packages (from nbstripout) (5.9.2)
Requirement already satisfied: fastjsonschema in /opt/conda/lib/python3.10/site-packages (from nbformat->nbstripout) (2.18.0)
Requirement already satisfied: jsonschema>=2.6 in /opt/conda/lib/python3.10/site-packages (from nbformat->nbstripout) (3.2.0)
Requirement already satisfied: jupyter-core in /opt/conda/lib/python3.10/site-packages (from nbformat->nbstripout) (5.3.1)
Requirement already satisfied: traitlets>=5.1 in /opt/conda/lib/python3.10/site-packages (from nbformat->nbstripout) (5.14.3)
Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.10/site-packages (from jsonschema>=2.6->nbformat->nbstripout) (23.2.0)
Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.10/site-packages (from jsonschema>=2.6->nbformat->nbstripout) (0.20.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.10/site-packages (from jsonschema>=2.6->nbformat->nbstripout) (70.0.0)
Requirement already satisfied: six>=1.11.0 in /opt/conda/lib/python3.10/site-packages (from jsonschema>=2.6->nbformat->nbstripout) (1.16.0)
Requirement already satisfied: platformdirs>=2.5 in /opt/conda/lib/python3.10/site-packages (from jupyter-core->nbformat->nbstripout) (3.10.0)
Downloading nbstripout-0.7.1-py2.py3-none-any.whl (15 kB)
WARNING: Error parsing requirements for pygments: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/Pygments-2.16.1.dist-info/METADATA'
WARNING: Error parsing requirements for asttokens: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/asttokens-2.2.1.dist-info/METADATA'
WARNING: Error parsing requirements for comm: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/comm-0.1.4.dist-info/METADATA'
WARNING: Error parsing requirements for exceptiongroup: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/exceptiongroup-1.1.3.dist-info/METADATA'
WARNING: Error parsing requirements for executing: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/executing-1.2.0.dist-info/METADATA'
WARNING: Error parsing requirements for importlib-metadata: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/importlib_metadata-6.8.0.dist-info/METADATA'
WARNING: Error parsing requirements for ipython: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/ipython-8.14.0.dist-info/METADATA'
WARNING: Error parsing requirements for jedi: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/jedi-0.19.0.dist-info/METADATA'
WARNING: Error parsing requirements for matplotlib-inline: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/matplotlib_inline-0.1.6.dist-info/METADATA'
WARNING: Error parsing requirements for parso: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/parso-0.8.3.dist-info/METADATA'
WARNING: Error parsing requirements for pexpect: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/pexpect-4.8.0.dist-info/METADATA'
WARNING: Error parsing requirements for prompt-toolkit: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/prompt_toolkit-3.0.39.dist-info/METADATA'
WARNING: Error parsing requirements for python-dateutil: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/python_dateutil-2.8.2.dist-info/METADATA'
WARNING: Error parsing requirements for pytz: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/pytz-2023.3.dist-info/METADATA'
WARNING: Error parsing requirements for stack-data: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/stack_data-0.6.2.dist-info/METADATA'
WARNING: Error parsing requirements for zipp: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/zipp-3.16.2.dist-info/METADATA'
Installing collected packages: nbstripout
WARNING: The script nbstripout is installed in '/home/jupyter/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed nbstripout-0.7.1
+ docker exec -u 0 jupyter-server /bin/bash -c '/etc/jupyter/scripts/extension/install_jupyter_contrib_nbextensions.sh && mkdir -p /home/jupyter/.jupyter/custom/ && cp /etc/jupyter/custom/google_sign_in.js /home/jupyter/.jupyter/custom/ && ls -la /etc/jupyter/custom/extension_entry_jupyter.js && cp /etc/jupyter/custom/extension_entry_jupyter.js /home/jupyter/.jupyter/custom/custom.js && cp /etc/jupyter/custom/safe-mode.js /home/jupyter/.jupyter/custom/ && cp /etc/jupyter/custom/edit-mode.js /home/jupyter/.jupyter/custom/ && mkdir -p /etc/jupyter/nbconfig'
Config option `kernel_spec_manager_class` not recognized by `EnableJupyterNbextensionsConfiguratorApp`.
Enabling: jupyter_nbextensions_configurator
- Writing config: /home/jupyter/.jupyter
- Validating...
jupyter_nbextensions_configurator 0.6.3 [32mOK[0m
Enabling notebook nbextension nbextensions_configurator/config_menu/main...
Enabling tree nbextension nbextensions_configurator/tree_tab/main...
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3061, in _dep_map
return self.__dep_map
File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2856, in __getattr__
raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_mapDuring handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3052, in _parsed_pkg_info
return self._pkg_info
File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2856, in __getattr__
raise AttributeError(attr)
AttributeError: _pkg_info. Did you mean: 'egg_info'?During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/bin/jupyter-contrib", line 8, in <module>
sys.exit(main())
File "/opt/conda/lib/python3.10/site-packages/jupyter_core/application.py", line 285, in launch_instance
return super().launch_instance(argv=argv, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/traitlets/config/application.py", line 1073, in launch_instance
app = cls.instance(**kwargs)
File "/opt/conda/lib/python3.10/site-packages/traitlets/config/configurable.py", line 583, in instance
inst = cls(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/jupyter_contrib_core/application.py", line 27, in __init__
self._refresh_subcommands()
File "/opt/conda/lib/python3.10/site-packages/jupyter_contrib_core/application.py", line 43, in _refresh_subcommands
get_subcommands_dict = entrypoint.load()
File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2473, in load
self.require(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2497, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 792, in resolve
new_requirements = dist.requires(req.extras)[::-1]
File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2779, in requires
dm = self._dep_map
File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3063, in _dep_map
self.__dep_map = self._compute_dependencies()
File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3072, in _compute_dependencies
for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3054, in _parsed_pkg_info
metadata = self.get_metadata(self.PKG_INFO)
File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1475, in get_metadata
value = self._get(path)
File "/opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1683, in _get
with open(path, 'rb') as stream:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/Pygments-2.16.1.dist-info/METADATA'
Or
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/opt/conda/lib/python3.10/site-packages/platformdirs-3.10.0.dist-info/METADATA'
At this point, I'm reinstalling 25 packages in my startup script, which would be workable, but that last one, platformdirs, won't resolve even by deleting and uninstalling the package. I don't know why this would be.
Additional details:
Terra account email address: archeney@ucsc.edu
Workspace Google Project ID: terra-037b9a34
Cluster ID:
Screenshot of cloud environment configuration:
Link to workspace: https://app.terra.bio/#workspaces/terra-brca-exchange/CERFAC
*I need to update to the most recent version of gnomad, as 0.64 has a bug where the gnomad version 4 exomes dataset cannot be accessed. The purpose of my project is to work with gnomad version4 data, so this is unfortunately not a case where I can just use the older version- I am completely blocked here.
(I did also attempt to update gnomad in the notebook instead, but that also fails when installing a dependency due to what I think is an unrelated issue.)
Thank you!
Comments
3 comments
Hi Allison,
Thank you for writing in about this issue. Can you share the workspace where you are seeing this issue with Terra Support by clicking the Share button in your workspace? The Share option is in the three-dots menu at the top-right.
Please provide us with a link to your workspace. We’ll be happy to take a closer look as soon as we can!
Kind regards,
Anthony
I have toggled it to "share"! Thank you!
Hi,
Did I share it successfully? Or might there be an update on this? I would be grateful even to hear ideas for other things to try to get it to work. Thanks!
Please sign in to leave a comment.