pynsist and tkinter

[arkwave]

Hello all, I'm hoping I can get some help on this issue here since Googling didn't return much in the way of answers. I'm trying to compile a GUI written with tkinter into an executable file. The directory structure is as follows:

simulator
|_ HistoricSimulator
|_ sim_ui
|_ graphix.py
|_ installer.cfg

graphix.py contains the code that opens/runs the GUI itself, while sim_ui and HistoricSimulator are libraries that contain a variety of helper functions used in the GUI.

My installer.cfg file looks like this:

[Application]
name=Historic_Simulator
version=1.0
entry_point=volgraphix:main
icon=sim_ui/logo.ico

[Python]
version=3.6.2

[Include]
packages= numpy
	pandas
	scipy
	sys
	os
	tkinter
	_tkinter
	copy
	collections
	matplotlib
	cycler
	six
	dateutil
	pyparsing
	sqlalchemy
	time
	ast
	plotly
	datetime
	math
	time
	timeit
	operator
	pprint 

files = HistoricSimulator/
	sim_ui/

However, there seems to be an error with tkinter being compiled. Specifically, I get the following error when running python -i build/nsis/Historic_Simulator.launch.pyw:

Traceback (most recent call last):
  File "C:\Program Files\Historic_Simulator\Historic_Simulator.launch.pyw", line 30, in <module>
    from volgraphix import main
  File "C:\Program Files\Historic_Simulator\pkgs\volgraphix.py", line 13, in <module>
    from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg
  File "C:\Program Files\Historic_Simulator\pkgs\matplotlib\backends\backend_tkagg.py", line 6, in <module>
    from six.moves import tkinter as Tk
  File "C:\Program Files\Historic_Simulator\pkgs\six.py", line 92, in __get__
    result = self._resolve()
  File "C:\Program Files\Historic_Simulator\pkgs\six.py", line 115, in _resolve
    return _import_module(self.mod)
  File "C:\Program Files\Historic_Simulator\pkgs\six.py", line 82, in _import_module
    __import__(name)
  File "C:\Program Files\Historic_Simulator\pkgs\tkinter\__init__.py", line 36, in <module>
    import _tkinter # If this fails your Python may not be configured for Tk
ImportError: DLL load failed: The specified module could not be found.

Weirdly enough, removing the _tkinter line under Packages allows the program to run when run with python -i builds/nsis/Historic_Simulator.launch.pyw, but the application still fails to open after installation.

Any idea what the issue could be? Any help would be much appreciated!

[takluyver]

Have a look through issue #124, where @SecretShop worked out what you need to do for a Tkinter app. :-)

[takluyver]

I do mean to turn their findings into an example for easy reference, but I haven't had time yet - I'm on my way to a conference and messing around with 3 other side projects.

[arkwave]

@takluyver thanks for the reference, following the steps let me advance from one error to the next. Progress! ;)

The new error I'm getting is as follows:

 Traceback (most recent call last):
   File "C:\Program Files (x86)\Historic_Simulator\pkgs\numpy\core\__init__.py", line 16, in <module>
      from . import multiarray
  ImportError: cannot import name 'multiarray'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files (x86)\Historic_Simulator\Historic_Simulator.launch.pyw", line 30, in <module>
    from volgraphix import main
  File "C:\Program Files (x86)\Historic_Simulator\pkgs\volgraphix.py", line 11, in <module>
    import matplotlib
  File "C:\Program Files (x86)\Historic_Simulator\pkgs\matplotlib\__init__.py", line 122, in <module>
    from matplotlib.cbook import is_string_like, mplDeprecation, dedent, get_label
  File "C:\Program Files (x86)\Historic_Simulator\pkgs\matplotlib\cbook.py", line 32, in <module>
    import numpy as np
  File "C:\Program Files (x86)\Historic_Simulator\pkgs\numpy\__init__.py", line 142, in <module>
    from . import add_newdocs
  File "C:\Program Files (x86)\Historic_Simulator\pkgs\numpy\add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "C:\Program Files (x86)\Historic_Simulator\pkgs\numpy\lib\__init__.py", line 8, in <module>
    from .type_check import *
  File "C:\Program Files (x86)\Historic_Simulator\pkgs\numpy\lib\type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "C:\Program Files (x86)\Historic_Simulator\pkgs\numpy\core\__init__.py", line 26, in <module>
    raise ImportError(msg)
ImportError: 
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

Original error was: cannot import name 'multiarray'

Following some recommendations I found on StackOverflow and other places, I've re-installed numpy, but to no avail. I even tried specifying numpy under pypi_wheels rather than packages so that it could be built from scratch, but it didn't work. Have there been other issues with numpy in this vein?

EDIT: All this is being run with Python 3.6.2 via Anaconda on a Windows 10 64-bit PC.

[takluyver]

Did you get the same error when specifying numpy in pypi_wheels, or a different one? I would recommend using pypi_wheels where possible, because it automatically picks numpy for the right Python version and 32/64-bit.

[arkwave]

Exactly the same error. The same error is also thrown when i specify numpy==1.12.1 under pypi_wheels.

[takluyver]

OK. Working with that application (i.e. build with pypi_wheels = numpy==1.12.1, then installed), can you check:

1. When you build it, there should be a line like Downloading wheel: or Using cached wheel: in the log. This may be hard to see because NSIS produces a lot of output - you can run pynsist installer.cfg --no-makensis to stop before running NSIS. Find that line and see what it says.
2. After installing the application, look in the folder C:\Program Files (x86)\Historic_Simulator\pkgs\numpy\core\ . Is there any file there whose name starts with multiarray?

[arkwave]

1. Yes, numpy 1.12.1 was built using a cached wheel: Using cached wheel: numpy-1.12.1-cp36-none-win32.whl
2. Yes, there are two files: multiarray.cp36-win32.pyd and multiarray_tests.cp36-win32.pyd.

Also hey thanks for the prompt replies, they're much appreciated 😄

[takluyver]

No problem, just don't rely on replies normally being that prompt. I'm travelling at the moment, and this is an interesting problem to think about when I'm on a train or in a hotel. But I'll also disappear at times when there's other stuff to do. ;-)

Back to the question! The file multiarray.cp36-win32.pyd looks like the right one. But maybe it's used 64-bit Python in combination with a 32-bit wheel. On the computer where you built it, have a look for a directory like:

C:\Users\You\AppData\Local\pynsist\

You should see a zip file in there. Does its name end with -win32.zip or -amd64.zip?

[arkwave]

I actually have both files: python-3.6.2-embed-win32.zip as well as python-3.6.2-embed-amd64.zip. There's also a pypi/numpy/1.12.1 and pypi/numpy/1.13.1, each of which contains the respective wheels for the numpy versions.

EDIT: I suppose it's noteworthy that I'm explicitly setting bitness=32 in accordance with the post you linked earlier up in this thread.

UPDATE: So I'm actually receiving two different errors, based on how i run the .exe after compiling with pynsist. Installing/running the application directly results in the following error appearing in %APPDATA%/Historic_Simulator.launch.pyw.log:

Traceback (most recent call last):
  File "C:\Program Files (x86)\Historic_Simulator\Historic_Simulator.launch.pyw", line 30, in <module>
    from volgraphix import main
  File "C:\Program Files (x86)\Historic_Simulator\pkgs\volgraphix.py", line 13, in <module>
    from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg
  File "C:\Program Files (x86)\Historic_Simulator\pkgs\matplotlib\backends\backend_tkagg.py", line 6, in <module>
    from six.moves import tkinter as Tk
  File "C:\Program Files (x86)\Historic_Simulator\pkgs\six.py", line 92, in __get__
    result = self._resolve()
  File "C:\Program Files (x86)\Historic_Simulator\pkgs\six.py", line 115, in _resolve
    return _import_module(self.mod)
  File "C:\Program Files (x86)\Historic_Simulator\pkgs\six.py", line 82, in _import_module
    __import__(name)
  File "C:\Program Files (x86)\Historic_Simulator\pkgs\tkinter\__init__.py", line 36, in <module>
    import _tkinter # If this fails your Python may not be configured for Tk
ImportError: DLL load failed: %1 is not a valid Win32 application.

Running python build/nsis/Historic_Simulator.launch.pyw results in the same multiarray import error reported above.

[takluyver]

Try deleting those files and see which one it downloads when you build. You can also check the python.exe it installs. Stay.maxsize can reveal if it's 32 bit or 64 bit (search for more info, as I'm on my phone now)

…

On 29 Aug 2017 11:33 a.m., "Ananth Ravi Kumar" ***@***.***> wrote: I actually have both files: python-3.6.2-embed-win32.zip as well as python-3.6.2-embed-amd64.zip. There's also a pypi/numpy/1.12.1 and pypi/numpy/1.13.1, each of which contains the respective wheels for the numpy versions. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#125 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUA9ZHLkxpTWLISoZutz5BqAjfNksL7ks5sc9rdgaJpZM4PEeAO> .

[arkwave]

Thanks for all the help! I managed to figure it out. Here's the sequence of steps I had to go through:

1. Uninstall any previous-installed versions of the application.
2. Remove the bitness=32 requirement, since my tcl86t.dll and tk86t.dll files were 64-bit DLLs. This got rid of the numpy.multiarray-related import errors.
3. Append the following packages to the packages section:

   pandas dependencies: pytz, dateutil, setuptools, six
plotly dependencies: requests, pkg_resources, decorator, pprint.

4. Include the DLL files into the pkg directory in the installer.cfg file, using the same method described here
5. Run pynsist installer.cfg.

The GUI now opens up when the .exe file is run. Still having some issues with actually getting it to produce output for some reason, but the GUI itself seems to be up and running.

[arkwave]

Okay, running into another issue I'm not entirely sure how to debug; sorry for the flurry of messages over the past few days.

I've narrowed down the issue with the output generation to a single function that makes a few SQL queries using the sqlalchemy.create_engine method. The rest of the GUI's functionality works perfectly when data is simply read in rather than fetched from the server, so I'm guessing this is where the problem lies.

However, I tried debugging it by running C:\\Program Files\\Application>Python\\python.exe "Application.launch.pyw" as mentioned in the Uncaught Exceptions section of the documentation, but I keep getting an error saying "This app can't run on your PC". This is despite a Developer build being currently active on this PC.

Any idea what the issue could be? Does pynsist have any known issues with running sqlalchemy?

[Siecje]

What database engine are you using? SQLite3 in memory? SQLite3 on disk? It may not be able to write the database file.

If postgresql or mysql they will need to be running.

Is there a traceback? Or just a Window 10 dialog that says "This app can't run on your PC"? Or is that in the command prompt?

[arkwave]

@Siecje Currently the program uses a remote connection to a MySQL database to fetch data. So would the connection need to be made during the call to app.main() in the app.launch.pyw file?

And no there's no traceback, just the Windows 10 dialog.

[Siecje]

The remote connection should work. Have you tried just running the Python\python.exe in the install directory and making the connection?

Might need to do some print statement debugging to find out how far it gets before the Windows 10 dialog. I've never seen that dialog.

[arkwave]

Figured it out. The Windows dialog was being thrown because the python.exe file was corrupted for some reason. Uninstalling and rebuilding fixed that issue. The SQL error was caused because an sqlalchemy dependency, psycopg2, was not explicitly specified.

So in summary:

1. Gotta ensure that the DLLs used in Tkinter and the Python types are the same (i.e. either x86 or x64). These DLL files should be installed into the $INSTDIR\pkgs directory.
2. pandas requires six, setuptools, dateutils and pytz to be specified explicitly in installer.cfg.
3. plotly requires requests, pkg_resources, decorator, and pprint to be specified explicitly in installer.cfg.
4. sqlalchemy requires psycopg2 to be specified explicitly in installer.cfg.

Thanks again for all the help guys, and my apologies again for the many many notifications!

[Siecje]

Glad you got it working. !!! 🎉

Requiring psycopg2 is strange. psycopg2 should only be required if you are connecting to a postgresql database.

[arkwave]

Agreed, I'm not entirely sure what's going on there. Also, I'd be happy to write all this up as an example and submit a pull request, if that's alright? pynsist is substantially more accessible than cx_freeze, pyinstaller or py2exe in my opinion (especially given the fact that the latter two still don't support Python 3.6) and so some more examples to encourage use couldn't hurt 😃

[takluyver]

Thanks for the follow up. I'd definitely like an example for using Tkinter with bundled Python (which is the default from Python 3.6). I'm not sure how best to do that, though - whether to include the DLLs in the repo, or a script to download them from somewhere.

Points 2-4 are instances of the same sort of thing: you need to list all the packages that are used, including dependencies of dependencies. Maybe this could be clearer in the docs? I have also thought about making a tool where you can run the application and get a list of all the modules that were loaded.

[HHk666]

did you have any issue with pandas? it seems even with all the dependencies specified in installer.cfg the program still complains about missing numpy when importing pandas. I've tried various thing and am still unable to make it work

[takluyver]

What error are you seeing? And are you specifying numpy in packages or in pypi_wheels?