https://pantsbuild.org/ logo
Join Slack
Powered by
Hi, I don't know if this has been asked before, bu...
# general
m
Hi, I don't know if this has been asked before, but I cannot find any solution after searching. I want to add a particular package (prophet) as a requirement for my pants project. In order to be installed, this package requires building (some C++ files need to be compiled), and in the process of generating the 
requirements.pex
file, I've been getting the error 
ModuleNotFoundError: No module named 'numpy'
. I've added the necessary dependencies following "Requirements with undeclared dependencies" on this page, but I'm still not able to pass this failure. In pants, it seems all the requirements are going to be included in one process, while the package maintainer (see here and here) recommends multiple installing the dependencies first and then in the next stage build and install the package. In light of this, when building 
requirements.pex
, is there a way to specify the requirements in multiple stages? Any recommendation on how to make this work?
e
There is a workaround for this when using Pex directly: https://github.com/pantsbuild/pex/issues/1751#issuecomment-1118844403 In short, switch to this requirement string:
Copy code
git+<https://github.com/facebook/prophet@df411c91929e82bb8d2c0f3a8b758cacdd6e1e60#egg=prophet&subdirectory=python>
Unfortunately, Pants still doesn't support this requirement format. It gets in the way by trying to parse requirement strings and insisting they conform to https://peps.python.org/pep-0508/ even though the underlying resolver it uses in practice is Pip via Pex which adds a few more custom formats including this one.
m
I see, so at the moment there is no way to have this kind of requirement string in 
python_requirement()
, right?
e
Correct. There are 2 features Pants thwarts today by parsing requirements a individual PEP-508 strings: 1. It does not allow for requirements.txt format extensions Pip supports - namely env var expansion 2. It does not support VCS requirements where you need to access a project in a subdirectory of the repo It's item 2 that is preventing you from working around prophet issues. If the Python project were not in a subdirectory of the VCS you could use a standard PEP-508 VCS URL requirement string:
Copy code
prophet @ git+<https://github.com/facebook/prophet@df411c91929e82bb8d2c0f3a8b758cacdd6e1e60>
That doesn't work though since prophet is in the 
python
subdirectory of that repo. This is not an impossible feature for Pants but its also not simple; that's why it has not been worked on. We have limited time. If you're interested in seeing this feature added to Pants though you could file an feature request issue. We generally provide technical details / ideas / code pointers, etc. on the ticket depending on how much design work is needed. Some folks then get ambitious and tackle the feature themselves. We're always happy to have help!
Another avenue is to bang on the door @ https://github.com/facebook/prophet to see about them doing a release! The VCS trick works on a fixed up un-released version - last I looked anyhow.
They haven't released in 2 years so maybe they lost interest in the pivot to the metaverse. That actually seems pretty likely. This is probably a non-funded / underfunded project now.
Aha, they have released. They changed their PyPI name from fbprophet to prophet: https://pypi.org/project/prophet/1.1.1/ - re-reading your psot - you know that. I would have suspected the fix in https://github.com/facebook/prophet@df411c91929e82bb8d2c0f3a8b758cacdd6e1e60#egg=prophet&amp;subdirectory=python was in the most recent release. Looking a bit...
I'm trying a build now with 
pex -vvv prophet --no-wheel -o prophet.pex
and its running along but building prophet from source takes a long time! I'll report back, but if you could actually provide the error output you're getting after adding 
-ldebug
to your pants command line, that would likely shed more light.
m
Sure, this section seems to be the relevant one (I'm trying fbprophet==0.7.1, but can potentially switch to another version)
Copy code
21:40:35.12 [DEBUG] Running Building requirements.pex with 31 requirements: boto3-stubs[s3]==1.18.35, boto3==1.17.112, botocore==1.20.112, catboost==0.26.1, ciso8601==2.2.0, dask-gateway==0.9.0, dask[dataframe]==2021.7.2, distributed==2021.7.2, fbprophet==0.7.1, future==0.18.2, jellyfish==0.8.8, moto==1.3.14, numpy==1.20.3, pandas-profiling==2.4.0, pandas==1.3.5, parmap==1.5.2, protobuf==3.17.3, pyarrow==9.0.0, pydantic==1.7.4, pystan==2.19.1.1, python-json-logger==2.0.2, pyyaml==5.4.1, requests<3,>=2.2, scikit-learn==0.24.0, scipy==1.7.1, shap==0.31.0, simplejson==3.16.0, snowflake-sqlalchemy==1.3.4, torch==1.9.0, tqdm==4.62.2, types-requests==2.25.6 under semaphore with concurrency id: 15, and concurrency: 16
21:40:35.12 [DEBUG] Completed: setup_sandbox
21:40:35.13 [DEBUG] spawned local process as Some(31644) for Process { argv: ["/usr/local/Caskroom/miniconda/base/envs/ml-models/bin/python3.7", "./pex", "--tmpdir", ".tmp", "--jobs", "16", "--python-path", "/Users/indra/.pyenv/versions/2.7.18/bin:/Users/indra/.pyenv/versions/3.7.8/bin:/Users/indra/src/pynest/.afvenv/bin:/usr/local/opt/mysql@5.7/bin:/Users/indra/.rbenv/shims:/usr/local/Caskroom/miniconda/base/envs/ml-models/bin:/usr/local/Caskroom/miniconda/base/condabin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/munki:/Library/Apple/usr/bin:/Users/indra/.cargo/bin:/Users/indra/go/bin:/usr/local/opt/fzf/bin:/Users/indra/.foundry/bin", "--output-file", "requirements.pex", "--no-emit-warnings", "--manylinux", "manylinux2014", "--python", "/usr/local/Caskroom/miniconda/base/envs/ml-models/bin/python3.7", "--sources-directory=source_files", "boto3-stubs[s3]==1.18.35", "boto3==1.17.112", "botocore==1.20.112", "catboost==0.26.1", "ciso8601==2.2.0", "dask-gateway==0.9.0", "dask[dataframe]==2021.7.2", "distributed==2021.7.2", "fbprophet==0.7.1", "future==0.18.2", "jellyfish==0.8.8", "moto==1.3.14", "numpy==1.20.3", "pandas-profiling==2.4.0", "pandas==1.3.5", "parmap==1.5.2", "protobuf==3.17.3", "pyarrow==9.0.0", "pydantic==1.7.4", "pystan==2.19.1.1", "python-json-logger==2.0.2", "pyyaml==5.4.1", "requests<3,>=2.2", "scikit-learn==0.24.0", "scipy==1.7.1", "shap==0.31.0", "simplejson==3.16.0", "snowflake-sqlalchemy==1.3.4", "torch==1.9.0", "tqdm==4.62.2", "types-requests==2.25.6", "--constraints", "__constraints.txt", "--no-pypi", "--index=<https://pypi.org/simple/>", "--index=<https://pypi.cbhq.net/>", "--resolver-version", "pip-2020-resolver", "--layout", "packed"], env: {"CPPFLAGS": "", "LANG": "en_US.UTF-8", "LDFLAGS": "", "PATH": "/Users/indra/src/pynest/.afvenv/bin:/usr/local/opt/mysql@5.7/bin:/Users/indra/.rbenv/shims:/usr/local/Caskroom/miniconda/base/envs/ml-models/bin:/usr/local/Caskroom/miniconda/base/condabin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/munki:/Library/Apple/usr/bin:/Users/indra/.cargo/bin:/Users/indra/go/bin:/usr/local/opt/fzf/bin:/Users/indra/.foundry/bin", "PEX_IGNORE_RCFILES": "true", "PEX_ROOT": ".cache/pex_root"}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<3135c4d5894d36934ee8e6a7e1a98d4441fe9f8d112456b57f49001ef6da0ca7>, size_bytes: 334 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, input_files: DirectoryDigest { digest: Digest { hash: Fingerprint<3135c4d5894d36934ee8e6a7e1a98d4441fe9f8d112456b57f49001ef6da0ca7>, size_bytes: 334 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: [] }, output_files: {}, output_directories: {RelativePath("requirements.pex")}, timeout: None, execution_slot_variable: None, concurrency_available: 31, description: "Building requirements.pex with 31 requirements: boto3-stubs[s3]==1.18.35, boto3==1.17.112, botocore==1.20.112, catboost==0.26.1, ciso8601==2.2.0, dask-gateway==0.9.0, dask[dataframe]==2021.7.2, distributed==2021.7.2, fbprophet==0.7.1, future==0.18.2, jellyfish==0.8.8, moto==1.3.14, numpy==1.20.3, pandas-profiling==2.4.0, pandas==1.3.5, parmap==1.5.2, protobuf==3.17.3, pyarrow==9.0.0, pydantic==1.7.4, pystan==2.19.1.1, python-json-logger==2.0.2, pyyaml==5.4.1, requests<3,>=2.2, scikit-learn==0.24.0, scipy==1.7.1, shap==0.31.0, simplejson==3.16.0, snowflake-sqlalchemy==1.3.4, torch==1.9.0, tqdm==4.62.2, types-requests==2.25.6", level: Info, append_only_caches: {CacheName("pex_root"): RelativePath(".cache/pex_root")}, jdk_home: None, platform_constraint: Some(Macos_x86_64), cache_scope: Successful }
21:40:35.16 [DEBUG] Completed: pants.backend.python.util_rules.python_sources.prepare_python_sources
21:41:53.82 [INFO] Completed: Building requirements.pex with 31 requirements: boto3-stubs[s3]==1.18.35, boto3==1.17.112, botocore==1.20.112, catboost==0.26.1, ciso8601==2.2.0, dask-gateway==0.9.0, dask[dataframe]==2021.7.2, distri... (431 characters truncated)
21:41:53.82 [DEBUG] Completed: Scheduling: Building requirements.pex with 31 requirements: boto3-stubs[s3]==1.18.35, boto3==1.17.112, botocore==1.20.112, catboost==0.26.1, ciso8601==2.2.0, dask-gateway==0.9.0, dask[dataframe]==2021... (443 characters truncated)
21:41:53.82 [DEBUG] Completed: pants.backend.python.util_rules.pex.build_pex
21:41:53.83 [DEBUG] Completed: pants.backend.python.goals.pytest_runner.setup_pytest_for_target
21:41:53.83 [DEBUG] Completed: Run Pytest
21:41:53.83 [DEBUG] Completed: `test` goal
21:41:53.83 [DEBUG] computed 1 nodes in 89.425354 seconds. there are 37973 total nodes.
21:41:53.83 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building requirements.pex with 31 requirements: boto3-stubs[s3]==1.18.35, boto3==1.17.112, botocore==1.20.112, catboost==0.26.1, ciso8601==2.2.0, dask-gateway==0.9.0, dask[dataframe]==2021.7.2, distributed==2021.7.2, fbprophet==0.7.1, future==0.18.2, jellyfish==0.8.8, moto==1.3.14, numpy==1.20.3, pandas-profiling==2.4.0, pandas==1.3.5, parmap==1.5.2, protobuf==3.17.3, pyarrow==9.0.0, pydantic==1.7.4, pystan==2.19.1.1, python-json-logger==2.0.2, pyyaml==5.4.1, requests<3,>=2.2, scikit-learn==0.24.0, scipy==1.7.1, shap==0.31.0, simplejson==3.16.0, snowflake-sqlalchemy==1.3.4, torch==1.9.0, tqdm==4.62.2, types-requests==2.25.6' failed with exit code 1.
stdout:

stderr:
  ERROR: Command errored out with exit status 1:
   command: /Users/indra/.cache/pants/named_caches/pex_root/venvs/s/f5c6d381/venv/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/vt/090k9p9n1pq8x8p50qc9_z6m0000gp/T/process-executionos6wtn/.tmp/pip-req-build-9ifslfxc/setup.py'"'"'; __file__='"'"'/private/var/folders/vt/090k9p9n1pq8x8p50qc9_z6m0000gp/T/process-executionos6wtn/.tmp/pip-req-build-9ifslfxc/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/vt/090k9p9n1pq8x8p50qc9_z6m0000gp/T/process-executionos6wtn/.tmp/pip-wheel-u9whcbt_
       cwd: /private/var/folders/vt/090k9p9n1pq8x8p50qc9_z6m0000gp/T/process-executionos6wtn/.tmp/pip-req-build-9ifslfxc/
  Complete output (40 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib
  creating build/lib/fbprophet
  creating build/lib/fbprophet/stan_model
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/private/var/folders/vt/090k9p9n1pq8x8p50qc9_z6m0000gp/T/process-executionos6wtn/.tmp/pip-req-build-9ifslfxc/setup.py", line 149, in <module>
      long_description_content_type='text/markdown',
    File "/Users/indra/.cache/pants/named_caches/pex_root/venvs/s/f5c6d381/venv/lib/python3.7/site-packages/setuptools/__init__.py", line 169, in setup
      return distutils.core.setup(**attrs)
    File "/usr/local/Caskroom/miniconda/base/envs/ml-models/lib/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/local/Caskroom/miniconda/base/envs/ml-models/lib/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/local/Caskroom/miniconda/base/envs/ml-models/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/Users/indra/.cache/pants/named_caches/pex_root/venvs/s/f5c6d381/venv/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/usr/local/Caskroom/miniconda/base/envs/ml-models/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/local/Caskroom/miniconda/base/envs/ml-models/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/Caskroom/miniconda/base/envs/ml-models/lib/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/local/Caskroom/miniconda/base/envs/ml-models/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/local/Caskroom/miniconda/base/envs/ml-models/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/private/var/folders/vt/090k9p9n1pq8x8p50qc9_z6m0000gp/T/process-executionos6wtn/.tmp/pip-req-build-9ifslfxc/setup.py", line 48, in run
      build_models(target_dir)
    File "/private/var/folders/vt/090k9p9n1pq8x8p50qc9_z6m0000gp/T/process-executionos6wtn/.tmp/pip-req-build-9ifslfxc/setup.py", line 36, in build_models
      from fbprophet.models import StanBackendEnum
    File "/private/var/folders/vt/090k9p9n1pq8x8p50qc9_z6m0000gp/T/process-executionos6wtn/.tmp/pip-req-build-9ifslfxc/fbprophet/__init__.py", line 8, in <module>
      from fbprophet.forecaster import Prophet
    File "/private/var/folders/vt/090k9p9n1pq8x8p50qc9_z6m0000gp/T/process-executionos6wtn/.tmp/pip-req-build-9ifslfxc/fbprophet/forecaster.py", line 14, in <module>
      import numpy as np
  ModuleNotFoundError: No module named 'numpy'
e
Yeah, the fbprophet will never work. Please try prophet.
👌 1
m
I did try prophet==1.0.1 before with the same error, will try prophet==1.1.1
e
Ok, that works fine for me even simulating the case where none of the prophet wheels are compatible with your target interpreter. That was done like so:
Copy code
$ cat requirements.txt
--no-binary prophet
prophet
$ pex --resolver-version pip-2020-resolver -r requirements.txt -o prophet.pex
$ ./prophet.pex -c 'import prophet; print(prophet.__file__)'
Importing plotly failed. Interactive plots will not work.
/home/jsirois/.pex/installed_wheels/fdcee5f95513d5d17177f0f0513190f9f2910833d865e30af33b8723fc82785c/prophet-1.1.1-cp310-cp310-linux_x86_64.whl/prophet/__init__.py
m
Great, thanks!
6 Views
Open in Slack
PreviousNext
Powered by