Dynamic reloading/adding/removing/ of completions as FPATH (or XDG_DATA_DIR) change

[BronzeDeer]

What do you want?

MVP: Allow to re-invoke compinit either directly or through a helper during runtime to index new completions that got added to FPATH
Bonus: Add the ability to diff functions provided between the last state of the FPATH and the new FPATH and do selective adding of comparison functions
Bonus2: Full-on "declarative" sync between current state of FPATH and loaded comp functions (i.e. remove functions that dropped from the FPATH)
Bonus3: Respect XDG_DATA_DIR as source for additional completions and/or merge current state of XDG_DATA_DIR into FPATH as they change

Why do you want this?

I'm a huge fan of this plugin and it has been my main driver to move to zsh over something like fish.
I'm currently massively upping my shell game by also employing nix/direnv/nix-direnv to dynamically load extra executables onto my path as I enter and exit certain directories.

Sadly, even though nix packages typically contain an FPATH directory with completion functions, I have not found a good way to enable completions for new executables that get added. I could exec into a new zsh, but that would break one of the main benefits of direnv (dynamic change to a running shell by sourcing EXPORT diffs from subshell and likely quickly stack up into an overly deep process tree. I really don't want to miss the powerful autocomplete from this plugin, nor can I feasible install every command globaly, due to version collisions.

For now, I tried to hack together a very simple POC which takes care of updating the FPATH when entering nix-direnv shells, and just calling compinit manually. After a lot of debugging I realized that compinit had been tombstoned by this plugin.

Manually removing the tombstone works unfunction compinit && autoload +X compinit && compinit, but I fear that this will break zsh-autocomplete, since compinit might have been disabled for more than performance reasons. Is there a way of calling compinit again in a zsh-autocomplete friendly way? (As an mvp I'm only searching for how to patch my own shell, where I am always aware of whether zsh-autocomplete is loaded or not so I can use specialised logic for this case).

I'm still fairly new to the zsh compsys, but I'm happy to learn more and contribute a PR for this feature, but I think I need some guidance as to how it needs to slot into the overall architecture of zsh-autocomplete.

Who else would benefit from this?

Many zsh users of direnv and most if not all zsh users of nix-direnv

How should it work?

For the MVP, provide an idempotent function that can be called to reinit completions to the current state of the FPATH, preferrably matching the existing logic around compinit
For the "Bonuses": Given a diff of the FPATH or old and new FPATH, only partially update comp functions to avoid constantly reparsing FPATH

Given the following situation:

• zsh and zsh-autocomplete have been correctly initialized
• no executable named foo is on the path
• no function _foo is on the FPATH
• compsys has no registered entry for foo and _foo

When I perform the following steps:
1.a new executable foo is added to PATH,
1.an entry is added to the FPATH which contains _foo with

#compdef  foo
compdef _foo foo
# [...]

1. new function autocomplete:re_load_completions() is called

Then I expect the following to happen:

• Completions for foo start working

[BronzeDeer]

Here's my current naive and kindof brute-force POC, in case someone is interested

# Towards the bottom of my zshrc
_fpath_sync:hook(){
      # SAVE BASH/POSIX style FPATH, since direnv and nix work in bash subshells
      if [[ ! -v FPATH_SYNC_OLD_FPATH ]]; then
        # echo "FPATH_SYNC init"
        # Do no re-init the first time around
        FPATH_SYNC_OLD_FPATH="$FPATH"
      elif [[ "$FPATH_SYNC_OLD_FPATH" != "$FPATH" ]]; then

        echo "FPATH Changed!"

        # Allow us to restore the previous compinit, to be a good citizen
        functions -c compinit compinit_orig
        unfunction compinit
        # restore original compinit
        autoload +X compinit
        # do not write dumpfile, since we are likely working on a temporary FPATH
        compinit -D
        # restore original function
        functions -c compinit_orig compinit

        FPATH_SYNC_OLD_FPATH="$FPATH"
      fi
    }

    typeset -ag precmd_functions
    if (( ! ''${precmd_functions[(I)_fpath_sync:hook]} )); then
    # Add our hook last to go after _direnv_hook
      precmd_functions=($precmd_functions _fpath_sync:hook)
    fi
    typeset -ag chpwd_functions
    if (( ! ''${chpwd_functions[(I)_fpath_sync:hook]} )); then
      # Add our hook last to go after _direnv_hook
      chpwd_functions=($chpwd_functions _fpath_sync:hook)
    fi

For now, I have added the following shellHook to a testshell to produce the FPATH update

shellHook=''
  # Parse FPATH into lines, filter down to binaries from nix store, then filter to those that have zsh functions in attached to them, finally prepend them to FPATH
  export FPATH=`echo $PATH | tr ':' "\n" | grep -E '^/nix/store/.*/bin$' | xargs -I{} realpath -e "{}/../share/zsh/site-functions" 2>/dev/null | tr "\n" ':'`$FPATH
''

This seems to work at first glance, but I am sceptical that this doesn't break something in new and creative ways through the widget overriting that compinit does

[BronzeDeer]

As a small update: I have actually started working most of the zsh-autocomplete agnostic parts of this into it's own little plugin: https://github.com/BronzeDeer/zsh-completion-sync

Currently it does not yet play nice with zsh-autocomplete, since it does force usage of the builtin compinit, but it will load completions from XDG and even optionally from paths relative to the bin dirs in $PATH, covering the ergonomics of most of the nix-shell and nix-direnv cases.

I'm still investigating how to make it play the nices with zsh-autocomplete. Would be very happy for some pointers!

[BronzeDeer]

Another small update: I've put in rudimentary support for automatically discovering and re-initializing zsh-autocomplete in my plugin https://github.com/BronzeDeer/zsh-completion-sync . This now allows automatic reloads without breaking zsh-autocomplete. I'm still fine tuning the performance to make the shell start less heavy

[BronzeDeer]

After some time away, I've rethought how to handle detection and reloading in my plugin and redid it for version 0.2.0.
The integration now works smoothly and automatically if zsh-autocomplete is installed.

It still relies on idempotent re-initiliazition, which means there are still some issues. One major point is that configurations, are lost, which is rather jarring. This is consistent with this part of the configuration documentation

You can use Zsh's bindkey command, after loading Autocomplete

Since we are reloading the whole plugin, we would likely have to reapply atleast all relevant bindkey settings. I can take a shot at coding a workaround for that, but I'm not sure if a better solution would be to rework the way zsh-autocomplete handles config/bindkey loading. (And it likely depends on whether the problematic redefinitions come from ZAC istelf or from the call to compinit, which we obviously cannot affect)

[BronzeDeer]

After some time away, I've rethought how to handle detection and reloading in my plugin and redid it for version 0.2.0.
The integration now works smoothly and automatically if zsh-autocomplete is installed.

It still relies on idempotent re-initiliazition, which means there are still some issues. One major point is that configurations, are lost, which is rather jarring. This is consistent with this part of the configuration documentation

You can use Zsh's bindkey command, after loading Autocomplete

Since we are reloading the whole plugin, we would likely have to reapply atleast all relevant bindkey settings. I can take a shot at coding a workaround for that, but I'm not sure if a better solution would be to rework the way zsh-autocomplete handles config/bindkey loading. (And it likely depends on whether the problematic redefinitions come from ZAC istelf or from the call to compinit, which we obviously cannot affect)