https://pantsbuild.org/ logo
Join Slack
Powered by
Hey all, is there an explicit way I can tell the p...
# general
r
Hey all, is there an explicit way I can tell the pants builder to only pull a specific requests/urllib3? As of May 4th, I noticed our builds are failing due to the OpenSSL mismatch between 1.1.1+ and 1.1.01. We require at the least 
requests==2.28.2,<2.30
and 
urllib3==1.26.1,<2
but the recent change to where requests and urllib3 are puled from are breaking our builds. Is there a 
legacy
flag or something we can put onto our builds to ignore pulling 
requests>=2.30
and 
urllib3>=2
?
➕ 2
e
You can do something like this:
Copy code
echo "urllib3<2" > constraints.txt
And then, when installing Pants you need 
PIP_CONSTRAINT=constraints.txt pants ...
. You can add this line to a 
.pants.bootstrap
file (or 
.env
file if using the 
pants
launcher instead of the 
./pants
script) to avoid needing to remember the env var export:
Copy code
export PIP_CONSTRAINT=constraints.txt
Pants maintainers are looking at potentially introducing a more comprehensive fix, but this should help work around.
r
Hmm interesting, so I'm trying to understand this fully - we're trying to do a 
generate-lockfile
for this even exporting the constaints.txt as mentioned and running it via 
./pants generate-lockfile
from the script, yet we're still getting 
urllib3==2.0.2
even if we make a direct reference to it. In our 
pants.toml
we're explicitly stating the 
requirement_constraints = "constraints.txt"
and the version we have is 
urllib3==1.26.15
. Even adding a 
.pants.bootstrap
file as guided hasn't resolved the issue.
✅ 1
e
Your problem description has little detail. I assumed a Pants install issue (Pants itself depends on urllib3 indirectly). Can you get really detailed about your lock file setup?
Basically, can you provide the contents of 
pants.toml
and the prior lock file?
r
Yep, can do - one moment. Apologies, this isn't a pants install issue per say
pants.toml
Copy code
[GLOBAL]
pants_version = "2.10.0"
backend_packages = [
  'pants.backend.python',
  'pants.backend.python.lint.black',
  'pants.backend.python.lint.flake8',
  'pants.backend.python.typecheck.mypy',
]
pantsd_invalidation_globs.add = ["**/requirements.txt"]
use_deprecated_python_macros = false

[source]
marker_filenames = ["SOURCE_ROOT"]

[python]
interpreter_constraints = ["CPython==3.8.13"]
requirement_constraints = "constraints.txt"

[python-bootstrap]
search_path = ["<PYENV>", "/opt/python3"]

[test]
# Note - Pants strips external env vars, so any additional values must be added here, globally, or per test. <https://www.pantsbuild.org/docs/python-test-goal#setting-environment-variables>
# When boto3 is used in tests, it reads the local config file first. This doesn't work in CI. 
# To make it consistent for local and CI, disable it here by pointing to non-existent file and set the region to a default.
extra_env_vars = ["AWS_DEFAULT_REGION=us-east-1", "AWS_CONFIG_FILE=non-existent.txt"]

[pytest]
version = "pytest==7.0.1"
extra_requirements.add = [
  "pytest-mock==3.6.1",
  "requests-mock==1.8.0",
  "pytest-asyncio==0.15.1"
]
lockfile = "3rdparty/pytest_lockfile.txt"

[anonymous-telemetry]
enabled = false

[mypy]
args = "--namespace-packages --explicit-package-bases"
extra_type_stubs = ["types-requests==2.27.7", "types-chardet==4.0.3", "types-urllib3==1.26.7", "boto3-stubs==1.20.38"]
and generated lockfile:
Copy code
# This lockfile was autogenerated by Pants. To regenerate, run:
#
#    ./pants generate-lockfiles --resolve=pytest
#
# --- BEGIN PANTS LOCKFILE METADATA: DO NOT EDIT OR REMOVE ---
# {
#   "version": 2,
#   "valid_for_interpreter_constraints": [
#     "CPython==3.8.13"
#   ],
#   "generated_with_requirements": [
#     "pytest-asyncio==0.15.1",
#     "pytest-cov!=2.12.1,<3.1,>=2.12",
#     "pytest-mock==3.6.1",
#     "pytest==7.0.1",
#     "requests-mock==1.8.0"
#   ]
# }
# --- END PANTS LOCKFILE METADATA ---

atomicwrites==1.4.1; python_version >= "3.6" and python_full_version < "3.0.0" and sys_platform == "win32" or sys_platform == "win32" and python_version >= "3.6" and python_full_version >= "3.4.0"
attrs==23.1.0; python_version >= "3.7"
certifi==2023.5.7; python_version >= "3.7"
charset-normalizer==3.1.0; python_full_version >= "3.7.0" and python_version >= "3.7"
colorama==0.4.6; python_version >= "3.6" and python_full_version < "3.0.0" and sys_platform == "win32" or sys_platform == "win32" and python_version >= "3.6" and python_full_version >= "3.7.0"
coverage==7.2.5; python_version >= "3.7"
idna==3.4; python_version >= "3.7"
iniconfig==2.0.0; python_version >= "3.7"
packaging==23.1; python_version >= "3.7"
pluggy==1.0.0; python_version >= "3.6"
py==1.11.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6"
pytest-asyncio==0.15.1; python_version >= "3.6"
pytest-cov==3.0.0; python_version >= "3.6"
pytest-mock==3.6.1; python_version >= "3.6"
pytest==7.0.1; python_version >= "3.6"
requests-mock==1.8.0
requests==2.30.0; python_version >= "3.7"
six==1.16.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.3.0"
tomli==2.0.1; python_full_version <= "3.11.0a6" and python_version >= "3.7"
urllib3==2.0.2; python_version >= "3.7"
e
So this is your pytest lock file (and super old Pants). Did you try adding 
"urllib3<2"
to 
extra_requirements.add
?
I don't think 
requirement_constraints
is used with lockfiles.
But you're in very old territory there; so I'm unsure.
r
Correct, we've using 
pants 2.10.0
for the longest time, which is why we were confused when our builds started breaking with whatever change happened May 4th
e
Ok. Please report back on the 
extra_requirements.add
+ regen lock file.
r
Yep that seems to have done it, thanks for the input! Do you think it still might be necessary to have a 
.pants.bootstrap
with only:
Copy code
PIP_CONSTRAINT=constraints.txt
In it or do you think that's unnecessary?
e
That is likely necessary to install Pants fresh. If you use the old 
./pants
script to run Pants you can test this by 
mv ~/.cache/pants/setup ~/.cache/pants/setup.sav
and trying 
./pants
- it should fail without all that.
👍 1
You'd want to also check 
./pants
sources 
.pants.bootstrap
(look at the script). If not, you need a newer copy from https://github.com/pantsbuild/setup/
r
Ah yea looks like we're using an older 
PEX
version, specifically 
2.1.42
. I'm guessing I can just replace the script with what's latest from that url to get it situated.
With some tuning of course
e
Yeah, it should be a drop-in replacement. You can also try the new 
pants
binary, but that's probably too much change to mix in right now: https://www.pantsbuild.org/docs/installation
It is backwards compatible back to 1.26.x though.
r
I did see that as well, will have to take a deeper look at that once we can get past initial issues.
Hey John, wanted to circle back on this - did end up getting through one issue, but I noticed that we have a version discrepancy of 
setuptools
. We're using v58.0.2 in our 
constraints.txt
but I noticed in the latest PEX file there's this if statement 
"${staging_dir}/install/bin/pip" install --quiet -U pip "setuptools<58" && \
. Any recommendations on getting around this?
I noticed in the comments it's to support pants 1.x but we're on 
2.10.0
e
That is just for the Pants venv (for running Pants). It has (should have) nothing whatsoever to do with your own Python code Pants runs goals against.
Exactly as earlier, if you're hitting an error you'll probably need to be detail oriented to get good remote support. I.E. command line with full output and any other relevant details of your setup.
r
Gotcha, let me see if I can describe out everything we're doing in our workflow in order to give a better picture
So this is mainly failing from a CI/CD perspective at the moment. Every time we have a new build through jenkins we have a stage set up where pants gets setup and several commands pants are run through the process as defined here:
Copy code
environment {
  PANTS_CONFIG_FILES = "pants.ci.toml"
  PANTS_SETUP_CACHE = ".pants/setup"
  PANTS_WATCH_FILESYSTEM = "false"
  PANTS_PANTSD = "false"
  PRE_COMMIT_HOME = "${WORKSPACE}/.pre-commit"
}
steps {
  script {
    sh label: 'Init tflint', script: 'tflint --init'
    if (env.BRANCH_NAME == 'develop' || env.BRANCH_NAME == 'main' || env.CHANGE_TARGET == 'develop') {
      sh label: 'Run pre-commit linting', script: 'SKIP=pants-test,pants-fmt,pants-lint pre-commit run --all-files'
      sh label: 'Run Pants fmt', script: './pants fmt ::'
      sh label: 'Run Pants lint', script: './pants lint ::'
      sh label: 'Run all Pants unit tests', script: './pants test ::'
    } else {
      // An additional refspec needs to be added to Jenkins job config to make the git diff work:
      //  +refs/heads/main:refs/remotes/@{remote}/main
      sh label: 'Run pre-commit linting', script: 'SKIP=pants-test,pants-fmt,pants-lint pre-commit run'
      sh label: 'Run Pants fmt', script: './pants fmt --changed-since=origin/main --changed-dependees=direct'
      sh label: 'Run Pants lint', script: './pants lint --changed-since=origin/main --changed-dependees=direct'
      sh label: 'Run Pants unit tests on changes', script: './pants test --changed-since=origin/main --changed-dependees=direct'
    }
  }
As of right now, we're still using pants version 
2.10.0
in our toml and the constraints are being fed in through our 
.pants.bootstrap
with the export function. The error we're seeing right off the bat with this build stage is the following:
Copy code
[2023-05-11T18:26:26.607Z] + ./pants fmt --changed-since=origin/main --changed-dependees=direct

[2023-05-11T18:26:26.860Z] Bootstrapping Pants using /root/.pyenv/shims/python3.8

[2023-05-11T18:26:26.860Z] Creating the virtualenv PEX.

[2023-05-11T18:26:26.860Z] Downloading the Pex PEX.

[2023-05-11T18:26:28.241Z] SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.103/pex> verified.

[2023-05-11T18:26:43.706Z] Installing pantsbuild.pants==2.10.0 into a virtual environment at /home/jenkins/agent/workspace/vidence_feature_fix_pants_issues/.pants/setup/bootstrap-Linux-x86_64/2.10.0_py38

[2023-05-11T18:26:56.805Z] ERROR: Cannot install setuptools<58 because these package versions have conflicting dependencies.
We're seeing this with the latest 
pants
shell of 
PEX version 2.1.103
e
" these package versions..." And no actual listing of package versions?
r
Exactly, it's strange - moving to the new Pex shows none of the wheels we used to see with our old builds. It just defaults to the verbiage you see there.
e
What do you mean by "moving to new Pex"? Updating your 
./pants
script with the latest from https://github.com/pantsbuild/setup/?
r
That's correct, just dropping https://github.com/pantsbuild/setup/blob/gh-pages/pants as our replacement 
./pants
script. Our original one is from some time back
e
Ok, well ~obviously you can edit that script and check in the edits; so that's one very easy way forward.
Perhaps start by getting rid of --quiet, etc. But you can probably just kill the whole setuptools bit.
r
Hey John - hope all is well. So we've been tooling around with skie-pants and using the binary in order to build out pants - but we're encountering a weird issue. Currently - it's trying to pick up all requirements across our multiple 
requirements.txt
files (we have several since they're used in individual service docker containers) and we're seeing a dependency conflict where pants is trying to generate a lockfile based on two different versions of a dependency. Is there a way to explicitly tell pants to only look at the requirements.txt in a certain directory and not subdirectories?
Our 
pants.toml
is the following:
Copy code
[GLOBAL]
pants_version = "2.15.0"
backend_packages = [
  'pants.backend.python',
  'pants.backend.python.lint.black',
  'pants.backend.python.lint.flake8',
  'pants.backend.python.typecheck.mypy',
]
pantsd_invalidation_globs.add = ["**/requirements.txt"]
# use_deprecated_python_macros = false

[source]
marker_filenames = ["SOURCE_ROOT"]

[python]
interpreter_constraints = ["CPython==3.9.*"]
# requirement_constraints = "constraints.txt"
enable_resolves = true
default_resolve = "myresolve"

[python.resolves]
myresolve = "3rdparty/pytest_lockfile.txt"

[python-bootstrap]
# search_path = ["<PYENV>", "/opt/python3"]
search_path = ["<PATH>", "<PYENV>"]

[test]
# Note - Pants strips external env vars, so any additional values must be added here, globally, or per test. <https://www.pantsbuild.org/docs/python-test-goal#setting-environment-variables>
# When boto3 is used in tests, it reads the local config file first. This doesn't work in CI. 
# To make it consistent for local and CI, disable it here by pointing to non-existent file and set the region to a default.
extra_env_vars = ["AWS_DEFAULT_REGION=us-east-1", "AWS_CONFIG_FILE=non-existent.txt"]

[pytest]
version = "pytest==7.3.1"
extra_requirements.add = [
  "pytest-mock==3.6.1",
  "requests-mock==1.10.0",
  "pytest-asyncio==0.21.0"
]
lockfile = "3rdparty/pytest_lockfile.txt"

[anonymous-telemetry]
enabled = false

[mypy]
args = "--namespace-packages --explicit-package-bases"
extra_type_stubs = ["types-requests==2.30.0.0", "types-chardet==5.0.4.6", "types-urllib3==1.26.25.13", "boto3-stubs==1.26.135"]
e
Sorry for the delay in responding, you got me just as I left on a break there! To your lead-in: that's great, but do know that scie-pants just launches (via 
execv
) Pants (its done doing that in ~1ms); so, for bug reporting purposes, if you see any Pants output at all, you're firmly in Pants-land for bug reporting purposes. Pants looks for requirements globally when inferring dependencies or generating a lock. The position in the filesystem is not part of the calculations of what is in or out of the global requirements list. To carve that up into several different requirements lists you need to use multiple "resolves". Hopefully this documentation about declaring and using named resolves gets you on your way: https://www.pantsbuild.org/docs/python-third-party-dependencies#multiple-lockfiles. If not, speak up.
2 Views
Open in Slack
PreviousNext
Powered by