https://pantsbuild.org/ logo
Join Slack
Powered by
I have an issue I'm not sure how to debug. I try t...
# general
r
I have an issue I'm not sure how to debug. I try to run 
./pants test a/test_file.py
on my python/pytest repo and that call pulls a dozen of third-party dependencies, which looks "normal" to me. The problem is that the test doesn't exec. It takes around 80 seconds to build 
requirements.pex
and then seems to loop forever when trying to build 
pytest_runner.pex
before my machine goes out of memory. I'm not sure what I did wrong or how to debug that 😕
h
h
I believe that's only relevant when remote caching is used, but might be wrong
@ripe-cpu-85141 are other tests working? (to help debug)
r
I'm not using remote caching, I'm really a beginner with pants 🙂 It's difficult to say if the other tests are working as they more or less all pull the same set of 3rd party deps. But if I remove the 
python_requirements
targets, it goes through (the tests fails but 
./pants test
ends)
👍 1
h
r
I don't for the project but do for the tools. For example I have
Copy code
[pytest]
lockfile = ".pytest.lock"
in the 
pants.toml
file
should I try to add lockfiles for the project?
I went ahead and added lockfiles for my project. It seems I still have the issue. I see some lines like this one:
Copy code
pex: Vendoring pkg_resources from PreparedGit(key='setuptools', requirement='git+<https://github.com/pantsbuild/setuptools@3acb925dd708430aeaf197ea53ac8a752f7c1863#egg=setuptools>', rewrite=True, constrain=False, constraints=None) @ /home/nhaller/.cache/pants/named_caches/pex_root/installed_wheels/6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822/pex-2.1.90-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools
Unfortunately, that won't fly in my restricted network environment.
Also, I have a line in my requirements like this one:
Copy code
boto3-stubs[account,autoscaling,s3,organizations,cloudformation,sts,ec2,elb,elbv2,health,lambda,stepfunctions,securityhub,xray,cloudtrail,kms,iam,cloudwatch,logs,config,backup,ce,glue,athena,ecr,ssm,route53,guardduty,s3control,secretsmanager,rds,sqs,resourcegroupstaggingapi, transfer, sns, servicecatalog, sagemaker, efs, redshift, ram, quicksight,opensearch, mq, firehose, emr, elasticache, eks, ecs, dynamodb, batch, apigateway, acm, acm-pca, route53resolver, accessanalyzer, events, kinesis, appsync, emr-containers, kendra, auditmanager, inspector2, ds, synthetics]
It seems pants really don't like it. The 
./pants generate-lockfiles
takes way longer with that (~150 seconds) and it seems to be the line blocking 
pytest_runner.pex
building
basically, each word in the square bracket will pull another python package. Am I right to suppose it would be better for pants to have those directly listed like
Copy code
boto3-stubs
mypy-boto3-account
mypy-boto3-autoscaling
...
?
Ok, I'm lost now. if I run 
./pants test a/tests/test_something.py
a first time, it works almost instantly. Then I run the same command a second time, it endlessly builds pytext_runner.pex. Then if I ctrl+c and re-run for a third time, it works instantly again. If I rerun, endlessly builds pytext_runner.pex and so on and so on
h
Huh, that is weird
Are you seeing pantsd restarts interleaved there? That would be log lines like
Copy code
09:36:26.87 [INFO] Initializing scheduler...
09:36:28.24 [INFO] Scheduler initialized.
I don't think pants should care about an unsual number of extras, like in
Copy code
boto3-stubs[account,autoscaling,s3,organizations,cloudformation,sts,ec2,elb,elbv2,health,lambda,stepfunctions,securityhub,xray,cloudtrail,kms,iam,cloudwatch,logs,config,backup,ce,glue,athena,ecr,ssm,route53,guardduty,s3control,secretsmanager,rds,sqs,resourcegroupstaggingapi, transfer, sns, servicecatalog, sagemaker, efs, redshift, ram, quicksight,opensearch, mq, firehose, emr, elasticache, eks, ecs, dynamodb, batch, apigateway, acm, acm-pca, route53resolver, accessanalyzer, events, kinesis, appsync, emr-containers, kendra, auditmanager, inspector2, ds, synthetics]
but I will test this out
Assuming you actually need those extras?
Which Pants version are you on?
And yeah, a user lockfile is probably a good idea here
Unless I'm wrong, those "vendoring" lines are Pex talking about its own vendored requirements, it isn't getting them from those URLs at your build time, they are already baked in to it
Are you seeing it fail on those, or were you just noticing the debug logs?
r
So I have 
pants_version = "2.12.0+git5d31cdb5"
I need the extras but it just pulls some extra packages, so I "expended" them like I said before with the 
mypy3-boto3-account
and so on
I don't see errors about the vendored reqs, those are just debug lines
Oh, I can't say for pantsd restart: it now works all the time. I'm confused
h
What changed from when it didn't work to when it did, e.g., is this now with a lockfile?
and/or with the extras expanded?
I'm testing out creating a lockfile with those unexpanded extras to see if we have some pathology there
No, looks like we're able to create it, although maybe the pathology is that it interacts badly with some of your other dependencies
Can you post your full requirements.txt before and after expanding those extras?
3 Views
Open in Slack
PreviousNext
Powered by