Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compile option to build a single-file executable #9149

Open
2 tasks
PaxG opened this issue Dec 18, 2024 · 0 comments
Open
2 tasks

Compile option to build a single-file executable #9149

PaxG opened this issue Dec 18, 2024 · 0 comments
Labels
feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged.

Comments

@PaxG
Copy link

PaxG commented Dec 18, 2024

Describe the feature

When running the AWS CLI on a slow NFS network, any simple command takes a long time, depending on the NFS load.
A strace execution shows about 4000 files are read, mostly from botocore Python library.
The same AWS CLI installed on a local disk can show same execution in 0.5s, which can be 10x or 100x faster than the execution on a slow NFS disk.
There's already the option --with-install-type=portable-exe but this still stores the Python libraries in many small files.
The previous feature request #6388 was approximately similar but was solved differently with a containerization.
What I'm looking for is a new compilation option that would trigger the --onefile option of the pyinstaller so that the final "aws" executable would require less NFS accesses.

Use Case

Many small aws commands launched in a shell script makes the overall script slow if each command takes 5s-10s instead of 0.5s with a local disk installation. On the other hand, it's not practical to run local installations on all the hosts of the cluster.

Proposed Solution

The AWS CLI container is a workaround, but still there's a small overhead to always run a container. When I launch the same command on a slow NFS disk is takes between 5s and 10s, vs. 1.5s when executed from a container, and 0.5s when executed from a local disk installation.

Other Information

Peformance measurements:

  • on a slow NFS installation:
/usr/bin/time -p <network_path>/aws eks get-token --cluster-name mycluster | grep real
real 6.63
real 7.16
real 7.45
  • on a local disk, same host:
/usr/bin/time -p /tmp/aws-cli/v2/2.22.18/bin/aws eks get-token --cluster-name mycluster | grep real
real 0.55
real 0.54
real 0.54
  • with the aws cli container:
/usr/bin/time -p apptainer exec aws.sif aws eks get-token --cluster-name mycluster | grep real
real 1.56
real 1.46
real 1.48

Filesystem accesses:

% strace -s 200 -f -t <network_path>/aws eks get-token --cluster-name mycluster | grep real > & strace.log
% grep "open <network_path>" strace.log | wc -l
3907

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

CLI version used

2.22.18

Environment details (OS name and version, etc.)

RHEL8

@PaxG PaxG added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged.
Projects
None yet
Development

No branches or pull requests

1 participant