https://pantsbuild.org/ logo
#general
Title
# general
r

rhythmic-airport-91451

05/08/2023, 7:28 PM
Hey all, is there an explicit way I can tell the pants builder to only pull a specific requests/urllib3? As of May 4th, I noticed our builds are failing due to the OpenSSL mismatch between 1.1.1+ and 1.1.01. We require at the least
requests==2.28.2,<2.30
and
urllib3==1.26.1,<2
but the recent change to where requests and urllib3 are puled from are breaking our builds. Is there a
legacy
flag or something we can put onto our builds to ignore pulling
requests>=2.30
and
urllib3>=2
?
2
e

enough-analyst-54434

05/09/2023, 7:42 PM
You can do something like this:
Copy code
echo "urllib3<2" > constraints.txt
And then, when installing Pants you need
PIP_CONSTRAINT=constraints.txt pants ...
. You can add this line to a
.pants.bootstrap
file (or
.env
file if using the
pants
launcher instead of the
./pants
script) to avoid needing to remember the env var export:
Copy code
export PIP_CONSTRAINT=constraints.txt
Pants maintainers are looking at potentially introducing a more comprehensive fix, but this should help work around.
r

rhythmic-airport-91451

05/09/2023, 8:20 PM
Hmm interesting, so I'm trying to understand this fully - we're trying to do a
generate-lockfile
for this even exporting the constaints.txt as mentioned and running it via
./pants generate-lockfile
from the script, yet we're still getting
urllib3==2.0.2
even if we make a direct reference to it. In our
pants.toml
we're explicitly stating the
requirement_constraints = "constraints.txt"
and the version we have is
urllib3==1.26.15
. Even adding a
.pants.bootstrap
file as guided hasn't resolved the issue.
1
e

enough-analyst-54434

05/09/2023, 8:23 PM
Your problem description has little detail. I assumed a Pants install issue (Pants itself depends on urllib3 indirectly). Can you get really detailed about your lock file setup?
Basically, can you provide the contents of
pants.toml
and the prior lock file?
r

rhythmic-airport-91451

05/09/2023, 8:25 PM
Yep, can do - one moment. Apologies, this isn't a pants install issue per say
pants.toml
Copy code
[GLOBAL]
pants_version = "2.10.0"
backend_packages = [
  'pants.backend.python',
  'pants.backend.python.lint.black',
  'pants.backend.python.lint.flake8',
  'pants.backend.python.typecheck.mypy',
]
pantsd_invalidation_globs.add = ["**/requirements.txt"]
use_deprecated_python_macros = false

[source]
marker_filenames = ["SOURCE_ROOT"]

[python]
interpreter_constraints = ["CPython==3.8.13"]
requirement_constraints = "constraints.txt"

[python-bootstrap]
search_path = ["<PYENV>", "/opt/python3"]

[test]
# Note - Pants strips external env vars, so any additional values must be added here, globally, or per test. <https://www.pantsbuild.org/docs/python-test-goal#setting-environment-variables>
# When boto3 is used in tests, it reads the local config file first. This doesn't work in CI. 
# To make it consistent for local and CI, disable it here by pointing to non-existent file and set the region to a default.
extra_env_vars = ["AWS_DEFAULT_REGION=us-east-1", "AWS_CONFIG_FILE=non-existent.txt"]

[pytest]
version = "pytest==7.0.1"
extra_requirements.add = [
  "pytest-mock==3.6.1",
  "requests-mock==1.8.0",
  "pytest-asyncio==0.15.1"
]
lockfile = "3rdparty/pytest_lockfile.txt"

[anonymous-telemetry]
enabled = false

[mypy]
args = "--namespace-packages --explicit-package-bases"
extra_type_stubs = ["types-requests==2.27.7", "types-chardet==4.0.3", "types-urllib3==1.26.7", "boto3-stubs==1.20.38"]
and generated lockfile:
Copy code
# This lockfile was autogenerated by Pants. To regenerate, run:
#
#    ./pants generate-lockfiles --resolve=pytest
#
# --- BEGIN PANTS LOCKFILE METADATA: DO NOT EDIT OR REMOVE ---
# {
#   "version": 2,
#   "valid_for_interpreter_constraints": [
#     "CPython==3.8.13"
#   ],
#   "generated_with_requirements": [
#     "pytest-asyncio==0.15.1",
#     "pytest-cov!=2.12.1,<3.1,>=2.12",
#     "pytest-mock==3.6.1",
#     "pytest==7.0.1",
#     "requests-mock==1.8.0"
#   ]
# }
# --- END PANTS LOCKFILE METADATA ---

atomicwrites==1.4.1; python_version >= "3.6" and python_full_version < "3.0.0" and sys_platform == "win32" or sys_platform == "win32" and python_version >= "3.6" and python_full_version >= "3.4.0"
attrs==23.1.0; python_version >= "3.7"
certifi==2023.5.7; python_version >= "3.7"
charset-normalizer==3.1.0; python_full_version >= "3.7.0" and python_version >= "3.7"
colorama==0.4.6; python_version >= "3.6" and python_full_version < "3.0.0" and sys_platform == "win32" or sys_platform == "win32" and python_version >= "3.6" and python_full_version >= "3.7.0"
coverage==7.2.5; python_version >= "3.7"
idna==3.4; python_version >= "3.7"
iniconfig==2.0.0; python_version >= "3.7"
packaging==23.1; python_version >= "3.7"
pluggy==1.0.0; python_version >= "3.6"
py==1.11.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6"
pytest-asyncio==0.15.1; python_version >= "3.6"
pytest-cov==3.0.0; python_version >= "3.6"
pytest-mock==3.6.1; python_version >= "3.6"
pytest==7.0.1; python_version >= "3.6"
requests-mock==1.8.0
requests==2.30.0; python_version >= "3.7"
six==1.16.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.3.0"
tomli==2.0.1; python_full_version <= "3.11.0a6" and python_version >= "3.7"
urllib3==2.0.2; python_version >= "3.7"
e

enough-analyst-54434

05/09/2023, 8:30 PM
So this is your pytest lock file (and super old Pants). Did you try adding
"urllib3<2"
to
extra_requirements.add
?
I don't think
requirement_constraints
is used with lockfiles.
But you're in very old territory there; so I'm unsure.
r

rhythmic-airport-91451

05/09/2023, 8:40 PM
Correct, we've using
pants 2.10.0
for the longest time, which is why we were confused when our builds started breaking with whatever change happened May 4th
e

enough-analyst-54434

05/09/2023, 8:51 PM
Ok. Please report back on the
extra_requirements.add
+ regen lock file.
r

rhythmic-airport-91451

05/09/2023, 8:54 PM
Yep that seems to have done it, thanks for the input! Do you think it still might be necessary to have a
.pants.bootstrap
with only:
Copy code
PIP_CONSTRAINT=constraints.txt
In it or do you think that's unnecessary?
e

enough-analyst-54434

05/09/2023, 8:56 PM
That is likely necessary to install Pants fresh. If you use the old
./pants
script to run Pants you can test this by
mv ~/.cache/pants/setup ~/.cache/pants/setup.sav
and trying
./pants
- it should fail without all that.
👍 1
You'd want to also check
./pants
sources
.pants.bootstrap
(look at the script). If not, you need a newer copy from https://github.com/pantsbuild/setup/
r

rhythmic-airport-91451

05/09/2023, 9:07 PM
Ah yea looks like we're using an older
PEX
version, specifically
2.1.42
. I'm guessing I can just replace the script with what's latest from that url to get it situated.
With some tuning of course
e

enough-analyst-54434

05/09/2023, 9:09 PM
Yeah, it should be a drop-in replacement. You can also try the new
pants
binary, but that's probably too much change to mix in right now: https://www.pantsbuild.org/docs/installation
It is backwards compatible back to 1.26.x though.
r

rhythmic-airport-91451

05/09/2023, 9:10 PM
I did see that as well, will have to take a deeper look at that once we can get past initial issues.
Hey John, wanted to circle back on this - did end up getting through one issue, but I noticed that we have a version discrepancy of
setuptools
. We're using v58.0.2 in our
constraints.txt
but I noticed in the latest PEX file there's this if statement
"${staging_dir}/install/bin/pip" install --quiet -U pip "setuptools<58" && \
. Any recommendations on getting around this?
I noticed in the comments it's to support pants 1.x but we're on
2.10.0
e

enough-analyst-54434

05/11/2023, 6:50 PM
That is just for the Pants venv (for running Pants). It has (should have) nothing whatsoever to do with your own Python code Pants runs goals against.
Exactly as earlier, if you're hitting an error you'll probably need to be detail oriented to get good remote support. I.E. command line with full output and any other relevant details of your setup.
r

rhythmic-airport-91451

05/11/2023, 6:54 PM
Gotcha, let me see if I can describe out everything we're doing in our workflow in order to give a better picture
So this is mainly failing from a CI/CD perspective at the moment. Every time we have a new build through jenkins we have a stage set up where pants gets setup and several commands pants are run through the process as defined here:
Copy code
environment {
  PANTS_CONFIG_FILES = "pants.ci.toml"
  PANTS_SETUP_CACHE = ".pants/setup"
  PANTS_WATCH_FILESYSTEM = "false"
  PANTS_PANTSD = "false"
  PRE_COMMIT_HOME = "${WORKSPACE}/.pre-commit"
}
steps {
  script {
    sh label: 'Init tflint', script: 'tflint --init'
    if (env.BRANCH_NAME == 'develop' || env.BRANCH_NAME == 'main' || env.CHANGE_TARGET == 'develop') {
      sh label: 'Run pre-commit linting', script: 'SKIP=pants-test,pants-fmt,pants-lint pre-commit run --all-files'
      sh label: 'Run Pants fmt', script: './pants fmt ::'
      sh label: 'Run Pants lint', script: './pants lint ::'
      sh label: 'Run all Pants unit tests', script: './pants test ::'
    } else {
      // An additional refspec needs to be added to Jenkins job config to make the git diff work:
      //  +refs/heads/main:refs/remotes/@{remote}/main
      sh label: 'Run pre-commit linting', script: 'SKIP=pants-test,pants-fmt,pants-lint pre-commit run'
      sh label: 'Run Pants fmt', script: './pants fmt --changed-since=origin/main --changed-dependees=direct'
      sh label: 'Run Pants lint', script: './pants lint --changed-since=origin/main --changed-dependees=direct'
      sh label: 'Run Pants unit tests on changes', script: './pants test --changed-since=origin/main --changed-dependees=direct'
    }
  }
As of right now, we're still using pants version
2.10.0
in our toml and the constraints are being fed in through our
.pants.bootstrap
with the export function. The error we're seeing right off the bat with this build stage is the following:
Copy code
[2023-05-11T18:26:26.607Z] + ./pants fmt --changed-since=origin/main --changed-dependees=direct

[2023-05-11T18:26:26.860Z] Bootstrapping Pants using /root/.pyenv/shims/python3.8

[2023-05-11T18:26:26.860Z] Creating the virtualenv PEX.

[2023-05-11T18:26:26.860Z] Downloading the Pex PEX.

[2023-05-11T18:26:28.241Z] SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.103/pex> verified.

[2023-05-11T18:26:43.706Z] Installing pantsbuild.pants==2.10.0 into a virtual environment at /home/jenkins/agent/workspace/vidence_feature_fix_pants_issues/.pants/setup/bootstrap-Linux-x86_64/2.10.0_py38

[2023-05-11T18:26:56.805Z] ERROR: Cannot install setuptools<58 because these package versions have conflicting dependencies.
We're seeing this with the latest
pants
shell of
PEX version 2.1.103
e

enough-analyst-54434

05/11/2023, 7:20 PM
" these package versions..." And no actual listing of package versions?
r

rhythmic-airport-91451

05/11/2023, 7:22 PM
Exactly, it's strange - moving to the new Pex shows none of the wheels we used to see with our old builds. It just defaults to the verbiage you see there.
e

enough-analyst-54434

05/11/2023, 7:39 PM
What do you mean by "moving to new Pex"? Updating your
./pants
script with the latest from https://github.com/pantsbuild/setup/?
r

rhythmic-airport-91451

05/11/2023, 7:40 PM
That's correct, just dropping https://github.com/pantsbuild/setup/blob/gh-pages/pants as our replacement
./pants
script. Our original one is from some time back
e

enough-analyst-54434

05/11/2023, 7:41 PM
Ok, well ~obviously you can edit that script and check in the edits; so that's one very easy way forward.
Perhaps start by getting rid of --quiet, etc. But you can probably just kill the whole setuptools bit.
r

rhythmic-airport-91451

05/18/2023, 4:34 PM
Hey John - hope all is well. So we've been tooling around with skie-pants and using the binary in order to build out pants - but we're encountering a weird issue. Currently - it's trying to pick up all requirements across our multiple
requirements.txt
files (we have several since they're used in individual service docker containers) and we're seeing a dependency conflict where pants is trying to generate a lockfile based on two different versions of a dependency. Is there a way to explicitly tell pants to only look at the requirements.txt in a certain directory and not subdirectories?
Our
pants.toml
is the following:
Copy code
[GLOBAL]
pants_version = "2.15.0"
backend_packages = [
  'pants.backend.python',
  'pants.backend.python.lint.black',
  'pants.backend.python.lint.flake8',
  'pants.backend.python.typecheck.mypy',
]
pantsd_invalidation_globs.add = ["**/requirements.txt"]
# use_deprecated_python_macros = false

[source]
marker_filenames = ["SOURCE_ROOT"]

[python]
interpreter_constraints = ["CPython==3.9.*"]
# requirement_constraints = "constraints.txt"
enable_resolves = true
default_resolve = "myresolve"

[python.resolves]
myresolve = "3rdparty/pytest_lockfile.txt"

[python-bootstrap]
# search_path = ["<PYENV>", "/opt/python3"]
search_path = ["<PATH>", "<PYENV>"]

[test]
# Note - Pants strips external env vars, so any additional values must be added here, globally, or per test. <https://www.pantsbuild.org/docs/python-test-goal#setting-environment-variables>
# When boto3 is used in tests, it reads the local config file first. This doesn't work in CI. 
# To make it consistent for local and CI, disable it here by pointing to non-existent file and set the region to a default.
extra_env_vars = ["AWS_DEFAULT_REGION=us-east-1", "AWS_CONFIG_FILE=non-existent.txt"]

[pytest]
version = "pytest==7.3.1"
extra_requirements.add = [
  "pytest-mock==3.6.1",
  "requests-mock==1.10.0",
  "pytest-asyncio==0.21.0"
]
lockfile = "3rdparty/pytest_lockfile.txt"

[anonymous-telemetry]
enabled = false

[mypy]
args = "--namespace-packages --explicit-package-bases"
extra_type_stubs = ["types-requests==2.30.0.0", "types-chardet==5.0.4.6", "types-urllib3==1.26.25.13", "boto3-stubs==1.26.135"]
e

enough-analyst-54434

06/01/2023, 2:02 PM
Sorry for the delay in responding, you got me just as I left on a break there! To your lead-in: that's great, but do know that scie-pants just launches (via
execv
) Pants (its done doing that in ~1ms); so, for bug reporting purposes, if you see any Pants output at all, you're firmly in Pants-land for bug reporting purposes. Pants looks for requirements globally when inferring dependencies or generating a lock. The position in the filesystem is not part of the calculations of what is in or out of the global requirements list. To carve that up into several different requirements lists you need to use multiple "resolves". Hopefully this documentation about declaring and using named resolves gets you on your way: https://www.pantsbuild.org/docs/python-third-party-dependencies#multiple-lockfiles. If not, speak up.
2 Views