Audit sysimage for relocatability #698

IanButterworth · 2022-06-14T21:40:04Z

It's finding absolute paths to the depot:

[ Info: Auditing sysimage relocatability
┌ Warning: absolute path `/tmp/jl_RgjRka` found in 104 places:
│ [27252233] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src/Preferences.jl
│ [27253801] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src/Preferences.jl
│ [27255433] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src/Preferences.jl
│ [27257113] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src/Preferences.jl
│ [27262345] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src/Preferences.jl
│ [27264217] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src/Preferences.jl
│ [27265673] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src/Preferences.jl
│ [27267193] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src/Preferences.jl
│ [27269049] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src/Preferences.jl
│ [28900313] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/toplevel_generators.jl
│ [28901705] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/runtime.jl
│ [28903529] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/toplevel_generators.jl
│ [28905177] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/toplevel_generators.jl
│ [28906809] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/toplevel_generators.jl
│ [28908425] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/toplevel_generators.jl
│ [28910073] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/wrapper_generators.jl
│ [28927177] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/runtime.jl
│ [55969481] /tmp/jl_RgjRka/packages/Example/aqsx3/src/Example.jl
│ [55970937] /tmp/jl_RgjRka/packages/Example/aqsx3/src/Example.jl
│ [69063929] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/toplevel_generators.jl
│ [69544249] /tmp/jl_RgjRka/packages/LLVMExtra_jll/RuRRc/Artifacts.toml
│ [69553609] /tmp/jl_RgjRka/packages/LLVMExtra_jll/RuRRc/src/wrappers/x86_64-linux-gnu-cxx11-llvm_version+11.jl
│ [78159609] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/toplevel_generators.jl
│ [78161449] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/products/executable_generators.jl
│ [78162601] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/products/executable_generators.jl
│ [78164729] /tmp/jl_RgjRka/packages/HelloWorldC_jll/lE1RA/Artifacts.toml
│ [78201145] /tmp/jl_RgjRka/packages/HelloWorldC_jll/lE1RA/src/wrappers/x86_64-linux-gnu.jl
│ [80454153] /tmp/jl_RgjRka/compiled/v1.6/MyApp/6XOLX_uCLnI.ji
│ [80454729] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/JLLWrappers.jl
│ [80454809] /tmp/jl_RgjRka/compiled/v1.6/JLLWrappers/7Zgw7_uCLnI.ji
│ [80456121] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src/Preferences.jl
│ [80456201] /tmp/jl_RgjRka/compiled/v1.6/Preferences/pWSk8_uCLnI.ji
│ [80456585] /tmp/jl_RgjRka/compiled/v1.6/LibSSH2_jll/K6mup_uCLnI.ji
│ [80457257] /tmp/jl_RgjRka/compiled/v1.6/MbedTLS_jll/u5NEn_uCLnI.ji
│ [80458121] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src/Crayons.jl
│ [80458201] /tmp/jl_RgjRka/compiled/v1.6/Crayons/TXPcU_uCLnI.ji
│ [80458601] /tmp/jl_RgjRka/packages/Example/aqsx3/src/Example.jl
│ [80458681] /tmp/jl_RgjRka/compiled/v1.6/Example/lLvWP_jGpNZ.ji
│ [80458905] /tmp/jl_RgjRka/compiled/v1.6/Zlib_jll/xjq3Q_uCLnI.ji
│ [80459801] /tmp/jl_RgjRka/packages/HelloWorldC_jll/lE1RA/src/HelloWorldC_jll.jl
│ [80459897] /tmp/jl_RgjRka/compiled/v1.6/HelloWorldC_jll/rt8Uf_uCLnI.ji
│ [80460761] /tmp/jl_RgjRka/packages/LLVMExtra_jll/RuRRc/src/LLVMExtra_jll.jl
│ [80460857] /tmp/jl_RgjRka/compiled/v1.6/LLVMExtra_jll/R9OeX_uCLnI.ji
│ [126003311] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src/crayon.jl
│ [126005023] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src/crayon_stack.jl
│ [126005085] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src/test_prints.jl
│ [126005146] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src/downcasts.jl
│ [126069455] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src/macro.jl
│ [126074778] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src/precompile.jl
│ [126150209] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src/utils.jl
│ [126151224] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src/Preferences.jl
│ [126156016] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/JLLWrappers.jl
│ [126156081] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/toplevel_generators.jl
│ [126156154] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/products/library_generators.jl
│ [126156235] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/runtime.jl
│ [126156456] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/wrapper_generators.jl
│ [126156539] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/products/file_generators.jl
│ [126156650] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/products/executable_generators.jl
│ [126176667] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src/crayon_wrapper.jl
│ [126178562] /tmp/jl_RgjRka/packages/Example/aqsx3/src/Example.jl
│ [126183841] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src/logo.jl
│ [126184554] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src/consts.jl
│ [126184636] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src/Crayons.jl
│ [126190218] /tmp/jl_RgjRka/packages/LLVMExtra_jll/RuRRc/.pkg/platform_augmentation.jl
│ [126190336] /tmp/jl_RgjRka/packages/LLVMExtra_jll/RuRRc/src/wrappers/x86_64-linux-gnu-cxx11-llvm_version+11.jl
│ [126198621] /tmp/jl_RgjRka/packages/HelloWorldC_jll/lE1RA/src/wrappers/x86_64-linux-gnu.jl
│ [150906902] /tmp/jl_RgjRka/packages/HelloWorldC_jll/lE1RA/src/wrappers
│ [150906961] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/products
│ [150907016] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src
│ [150907262] /tmp/jl_RgjRka/packages/LLVMExtra_jll/RuRRc/src/wrappers
│ [150907319] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src/products
│ [150907374] /tmp/jl_RgjRka/packages/JLLWrappers/QpMQW/src
│ [150907646] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src
│ [150908398] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src
│ [150910062] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src
│ [150911069] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src
│ [150911177] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src
│ [150911782] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src
│ [150993253] /tmp/jl_RgjRka/packages/Example/aqsx3/src
│ [151013682] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src
│ [151014287] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src
│ [151014791] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src
│ [151015037] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src
│ [151015157] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src
│ [151015277] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src
│ [151015397] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src
│ [151015517] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src
│ [151015637] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src
│ [151015757] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src
│ [151015877] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src
│ [151015997] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src
│ [151026677] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src
│ [151026820] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src
│ [151027379] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src
│ [151069235] /tmp/jl_RgjRka/packages/Example/aqsx3/src
│ [154156331] /tmp/jl_RgjRka/packages/HelloWorldC_jll/lE1RA/src/wrappers/x86_64-linux-gnu.jl
│ [154156443] /tmp/jl_RgjRka/packages/LLVMExtra_jll/RuRRc/src/wrappers/x86_64-linux-gnu-cxx11-llvm_version+11.jl
│ [154156542] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src/Preferences.jl
│ [154156947] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src/precompile.jl
│ [154157613] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src/macro.jl
│ [154158072] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src/Crayons.jl
│ [154158125] /tmp/jl_RgjRka/packages/Crayons/u3AH8/src/crayon.jl
│ [154166046] /tmp/jl_RgjRka/packages/Example/aqsx3/src/Example.jl
│ [154168032] /tmp/jl_RgjRka/packages/Preferences/VmJXL/src/utils.jl
└ @ PackageCompiler ~/work/PackageCompiler.jl/PackageCompiler.jl/src/PackageCompiler.jl:1178

staticfloat · 2022-06-14T21:59:18Z

Nice. Some of these are doubtless the paths to .jl files that are stored in .ji files for invalidation. Since PackageCompiler usually doesn't bundle the source for packages, I think it might be best to filter out references to .jl source files.

staticfloat · 2022-06-14T21:59:51Z

Oh, but this is a .so, not a .ji. Hmmm, do those paths still get embedded?

codecov · 2022-06-14T22:28:40Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.88%. Comparing base (85cdb03) to head (78bd67a).

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #698      +/-   ##
==========================================
+ Coverage   84.56%   84.88%   +0.31%     
==========================================
  Files           3        3              
  Lines         823      840      +17     
==========================================
+ Hits          696      713      +17     
  Misses        127      127

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

KristofferC · 2022-06-15T08:18:20Z

https://julialang.github.io/PackageCompiler.jl/stable/apps.html#Absolute-paths-of-the-build-machine

IanButterworth · 2022-06-15T16:29:07Z

I'm happy to look into trying to fix this in Base etc.

Do you have any tips on where these paths might be stored?

KristofferC · 2022-06-15T17:23:45Z

The file field of a Method for example.

But there's really not a "correct answer" to this though since the files might not exist at all. So what are you going to put in there?

IanButterworth · 2022-06-15T17:31:53Z

Thanks.

I guess a first step is to make paths to the depot relative. Whether the file actually exists and what to do about that could come later?

staticfloat · 2022-06-15T17:36:15Z

For things that are obviously inside of a depot, we could replace them with ${DEPOT_PATH} or similar, and then provide substitution functions that search for ${DEPOT_PATH} and just sub in the path of the depot the object that was loaded came out of.

DilumAluthge · 2024-10-17T23:01:41Z

It would be nice to get this PR in, since it is just informational. I think that work on actually removing or fixing the absolute paths can happen in follow-up PRs, right?

src/PackageCompiler.jl

Co-authored-by: Dilum Aluthge <[email protected]>

src/PackageCompiler.jl

Co-authored-by: Dilum Aluthge <[email protected]>

IanButterworth · 2024-10-18T01:49:58Z

@DilumAluthge feel free to just push and take this over

DilumAluthge · 2024-10-19T23:46:12Z

I think this PR adds some good information. Also, this PR does not add the audit_relocatability kwarg to the public API. Therefore, if we merge this PR, we can revert it in the future without making a breaking change.

I'm inclined to merge this PR (since it just adds information), to see if people find this useful in their usage. @IanButterworth @staticfloat @KristofferC Any objection to merging this PR.

If there are no objections, I'll proceed with merging this PR and making a new release.

IanButterworth added 2 commits June 14, 2022 17:39

audit sysimage for relocatability

37b20cb

intentionally include an abs path for testing

65f395e

force print all paths

6e6dec0

IanButterworth force-pushed the ib/relocatability_audit branch from 22b04ab to 6e6dec0 Compare June 14, 2022 22:21

Merge branch 'master' into ib/relocatability_audit

5540419

Merge branch 'master' into ib/relocatability_audit

8cb4cd7

DilumAluthge reviewed Oct 18, 2024

View reviewed changes

src/PackageCompiler.jl Outdated Show resolved Hide resolved

Update PackageCompiler.jl

0e6471c

Co-authored-by: Dilum Aluthge <[email protected]>

DilumAluthge reviewed Oct 18, 2024

View reviewed changes

src/PackageCompiler.jl Outdated Show resolved Hide resolved

Update PackageCompiler.jl

78bd67a

Co-authored-by: Dilum Aluthge <[email protected]>

DilumAluthge marked this pull request as ready for review October 18, 2024 03:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Audit sysimage for relocatability #698

Audit sysimage for relocatability #698

IanButterworth commented Jun 14, 2022 •

edited by DilumAluthge

Loading

staticfloat commented Jun 14, 2022

staticfloat commented Jun 14, 2022

codecov bot commented Jun 14, 2022 •

edited

Loading

KristofferC commented Jun 15, 2022

IanButterworth commented Jun 15, 2022

KristofferC commented Jun 15, 2022

IanButterworth commented Jun 15, 2022

staticfloat commented Jun 15, 2022

DilumAluthge commented Oct 17, 2024

IanButterworth commented Oct 18, 2024

DilumAluthge commented Oct 19, 2024 •

edited

Loading

Audit sysimage for relocatability #698

Are you sure you want to change the base?

Audit sysimage for relocatability #698

Conversation

IanButterworth commented Jun 14, 2022 • edited by DilumAluthge Loading

staticfloat commented Jun 14, 2022

staticfloat commented Jun 14, 2022

codecov bot commented Jun 14, 2022 • edited Loading

Codecov Report

KristofferC commented Jun 15, 2022

IanButterworth commented Jun 15, 2022

KristofferC commented Jun 15, 2022

IanButterworth commented Jun 15, 2022

staticfloat commented Jun 15, 2022

DilumAluthge commented Oct 17, 2024

IanButterworth commented Oct 18, 2024

DilumAluthge commented Oct 19, 2024 • edited Loading

IanButterworth commented Jun 14, 2022 •

edited by DilumAluthge

Loading

codecov bot commented Jun 14, 2022 •

edited

Loading

DilumAluthge commented Oct 19, 2024 •

edited

Loading