神刀安全网

Creating an embeddable Python distribution on OS X

Creating an embeddable Python distribution on OS X

An (almost) complete tutorial

June 05, 2016

I am currently working on a cross-platform Electron application that needs to make network calls to an embedded Python web server. Since this application must run on OS X and Windows, and I don’t want my users to have to install Python 3 themselves, I have to include the Python interpreter with the application. Since Python 3.5 we can use the embeddable Python distribution on Windows, but there’s not such thing for OS X and we have to roll our own.

This blog post documents one of several possible ways of building a standalone Python distribution on OS X that you can use to embed in other applications. I know that there are solutions such as py2app or cx_freeze, but I’ve never had much success with them previously. Besides, I want to embed Python on another application, not create a standalone executable (although we can do it).

Install a working Python 3 distribution

We are going to build our standalone distribution from Homebrew ‘s Python distribution. You can probably use the official Python installer from Python.org, although I prefer to do brew install python3 on the terminal.

Homebrew will install Python 3 in /usr/local/Frameworks/Python.framework/Versions/3.5/ .

Creating an embeddable Python distribution on OS X

These are the contents of the directories:

  • bin : includes several programs such as the python3 and python3.5 binary programs and the pyvenv-3.5 to build virtual environments.
  • Headers : symbolic link to include .
  • include : includes the python development headers, necessary if we need to compile anything against Python.
  • lib : includes symlinks for the Python shared library (symlinks are named libpython3.5.dylib and libpython3.5m.dylib) and the python standard library inside lib/python3.5 .
  • Python : the python shared library.
  • Resources : includes the OS X Info.plist file and the Python launcher (Python.app).
  • share : documentation.

As a side note, if you couldn’t remember where Homebrew installed Python 3, here’s how you could find it:

MacBook-Pro$ which python3 /usr/local/bin/python3  MacBook-Pro$$ readlink /usr/local/bin/python3 ../Cellar/python3/3.5.1/bin/python3  MacBook-Pro$ readlink /usr/local/Cellar/python3/3.5.1/bin/python3 ../Frameworks/Python.framework/Versions/3.5/bin/python3

The first command shows us the location of the homebrew’s python3 command. Since it is just a symlink to another symlink, we use the readlink command to find the destination. We could have used the Finder as well..

Cherry pick some files

Create an empty folder named python3.5 somewhere (I like do it on my Desktop ). We are going to copy files from the Homebrew’s source (I’ll use the $homebrew prefix to refer to it) to build our Python distribution.

First we are going to copy the python shared lib to our destination. Copy the $homebrew/Python file and rename it to libpython3.5.dylib .

Next, copy the include/ and lib/ folders to the destination. You can delete lib/libpython3.5.dylib , lib/libpython3.5m.dylib and lib/pkgconfig as we won’t be using them. If you are not going to compile anything, you can also delete the include folder.

Creating an embeddable Python distribution on OS X

Congratulations, you now have a portable Python distribution! Unfortunately, you cannot do much with it as you don’t have any binary programs that use it.

Python Interactive Interpreter

So far, we have our distribution almost ready, but we cannot make use of it. In this section we are going to make use of the binary program which allows us to do things such as "python3.5 a_file.py" or just "python3.5" to launch the interactive interpreter.

For those with more interest in this kind of things, the source code for the python launcher can be found at https://github.com/python/cpython/blob/3.5/Programs/python.c . The most relevant line is where the application calls the main function from the dynamic library ( Py_Main(argc, argv_copy) ). In simple words, the python dynamic library does all the heavy work.

Although we could compile the launcher from the source code, we will use the binary from the homebrew distribution. Right click on $homebrew/Resources/Python.app and select "Show package contents". OS X apps are just plain folders. Copy the Contents/MacOSX/Python file and rename it to python3.5 .

Creating an embeddable Python distribution on OS X

Now, execute the file. You will be greeted with the python interactive interpreter:

MacBook-Pro$ ./python3.5 Python 3.5.1 (default, Apr 18 2016, 11:55:04) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>

Stop, you are being fooled!!!!!

Wanna see?

>>> import sys >>> sys.path ['', '/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python35.zip',  '/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5',  '/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/plat-darwin',  '/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/lib-dynload',  '/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages'] >>>

This program is using homebrew’s python distribution (/usr/local/Cellar/…) and not the one we have been building. If you were to distribute these files to a computer without homebrew or if you deleted your python 3, this would not work anymore. So how do we fix it?

First, let’s see which dynamic library the python3.5 binary is using:

MacBook-Pro$ otool -L python3.5 python3.5:    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 855.17.0)    /usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/Python (compatibility version 3.5.0, current version 3.5.0)    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

As you can see, this program is using the python dynamic library from the homebrew directory! Let’s change it to use libpython3.5.dylib from our distribution:

MacBook-Pro$ install_name_tool -change /usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/Python @executable_path/libpython3.5.dylib python3.5

Let’s run the otool command again:

MacBook-Pro$ otool -L python3.5 python3.5:     /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 855.17.0)     @executable_path/libpython3.5.dylib (compatibility version 3.5.0, current version 3.5.0)     /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

The program now seems to use "our" dynamic library. Let’s run the interactive interpreter to make sure:

MacBook-Pro$ ./python3.5 Python 3.5.1 (default, Apr 18 2016, 11:55:04) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path ['', '/Users/user/Desktop/python3.5/lib/python35.zip', '/Users/user/Desktop/python3.5/lib/python3.5', '/Users/user/Desktop/python3.5/lib/python3.5/plat-darwin', '/Users/user/Desktop/python3.5/lib/python3.5/lib-dynload'] >>>

Congratulations, now we have a workable portable python distribution!

Compressing the standard library

As you can see above, python also includes a python35.zip file in sys.path. This means that we can compress the standard library to save some space.

Open lib/python3.5/ , select all files and folders except lib-dynload (compiled modules) and plat-darwin (platform specific things) and add them to a zip file named python35.zip . Make sure that you create the zip file directly on lib/python3.5/ as you don’t want any sub-folder between the root of the zip file and the standard lib files. Move the file to lib/python35.zip .

This is how it should be like:

Creating an embeddable Python distribution on OS X

Run your local python3.5 file and everything should work as expected:

MacBook-Pro$ ./python3.5 Python 3.5.1 (default, Apr 18 2016, 11:55:04) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import this The Zen of Python, by Tim Peters  Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. Special cases aren't special enough to break the rules. Although practicality beats purity. Errors should never pass silently. Unless explicitly silenced. In the face of ambiguity, refuse the temptation to guess. There should be one-- and preferably only one --obvious way to do it. Although that way may not be obvious at first unless you're Dutch. Now is better than never. Although never is often better than *right* now. If the implementation is hard to explain, it's a bad idea. If the implementation is easy to explain, it may be a good idea. Namespaces are one honking great idea -- let's do more of those! >>>

Compiling a binary program

To end this post I will show how you could compile your own program to make use of this embedded python distribution that you made yourself. Let’s create a file named hello.c on the same directory with the following content:

#include "Python.h"  int main(int argc, char **argv) {     // Add the standard lib as zip to system path     // Lib-dynload must be included because of zlib (we could delete the rest if we wanted!)     wchar_t *stdlib = L"lib/python35.zip:lib/python3.5/lib-dynload";     Py_SetPath(stdlib);      // Initialize Python interpreter     printf("Initializing the Python interpreter/n");     Py_Initialize();      // Run something     PyRun_SimpleString("print('Hello World!')");      // Finalize     printf("Finalizing the Python interpreter/n");     Py_Finalize();      return 0; }

Note that we are including python35.zip and lib-dynload because we are following a kind-of tutorial and I left the standard library as zip from the previous section, but we could add lib/python3.5 if we still had the uncompressed files.

Let’s compile the file using the headers in include and our local shared library, and run it:

MacBook-Pro$ cc hello.c -o hello -I include/python3.5m/ -L ./ -l python3.5 MacBook-Pro$ ./hello Initializing the Python interpreter Hello World! Finalizing the Python interpreter

Stop, you are being fooled again!!!!!

The thing works but let’s use our otool friend again:

MacBook-Pro$ otool -L hello hello:     /usr/local/opt/python3/Frameworks/Python.framework/Versions/3.5/Python (compatibility version 3.5.0, current version 3.5.0)     /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

As you can see, our "hello" binary it is using a dynamic library on /usr/local/opt/ . If we were to use this in a computer without homebrew’s python3, it wouldn’t work anymore. There are two solutions for this:

One is to force the "hello" binary to use our libpython3.5.dylib file:

MacBook-Pro$ install_name_tool -change /usr/local/opt/python3/Frameworks/Python.framework/Versions/3.5/Python @executable_path/libpython3.5.dylib hello

The problem with this approach is that only works for this binary, or in other words, if you needed to recompile "hello.c" you would have to execute the same command again.

But what is the other alternative? Let’s use the otool command on libpython3.5.dylib:

MacBook-Pro$ otool -L libpython3.5.dylib libpython3.5.dylib:     /usr/local/opt/python3/Frameworks/Python.framework/Versions/3.5/Python (compatibility version 3.5.0, current version 3.5.0)     /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 855.17.0)     /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

There’s a /usr/local/opt which is quite suspect. Let’s check in more detail:

MacBook-Pro$ otool -l libpython3.5.dylib ... cmd LC_ID_DYLIB   cmdsize 96      name /usr/local/opt/python3/Frameworks/Python.framework/Versions/3.5/Python (offset 24) ... cmd LC_LOAD_DYLIB   cmdsize 104      name /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (offset 24) ... cmd LC_LOAD_DYLIB   cmdsize 56      name /usr/lib/libSystem.B.dylib (offset 24)

As you can see, while CoreFoundation and libSystem are being loaded (LC_LOAD_DYLIB), (usr/local/opt/…/Python) seems to be some kind of identifier (LC_ID_DYLIB). There’s not much information about LC_ID_DYLIB out there, but install_name_tool has an -id parameter that we can try:

MacBook-Pro$ chmod 777 libpython3.5.dylib MacBook-Pro$ install_name_tool -id @executable_path/libpython3.5.dylib libpython3.5.dylib  MacBook-Pro$ otool -l libpython3.5.dylib ... cmd LC_ID_DYLIB   cmdsize 64      name @executable_path/libpython3.5.dylib (offset 24)

It seems to have worked, let’s recompile "hello.c" and check the libraries used:

MacBook-Pro$ cc hello.c -o hello -I include/python3.5m/ -L ./ -l python3.5 MacBook-Pro$ otool -L hello hello:     @executable_path/libpython3.5.dylib (compatibility version 3.5.0, current version 3.5.0)     /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

Great, now our python dynamic library is automatically identified when we compile. The only thing I notice is that since we are using @executable_path , we must have the "hello" library in the same directory as libpython3.5.dylib. If we had to move it, we would have to use the install_name_tool to point to the location of our dynamic library. This guy explains better than me alternatives to @executable_path.

I’ve explored this in two days, so there’s probably some simpler ways of doing everything I talk about here. Leave a comment if you know about any simpler alternative..

Good luck!

转载本站任何文章请注明:转载至神刀安全网,谢谢神刀安全网 » Creating an embeddable Python distribution on OS X

分享到:更多 ()

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址