This is a rare one that I don't see very often (outside of installers), but it's fully possible to compile your C/C++/insert-language-here from the Python runtime, and executing it using ctypes external library. Assuming you know why you would need this, here's a short embedded "hello world" example that we will import as a shared library. This rather short program will import two helper libraries, tempfile is to get the temporary path defined by the system in which we'll place all our temprary files and code, the ctypes library to execute the compiled result, as well as the ccompiler helpers to actually compile the embedded source code. The new_compiler() helper is not strictly necessary, but it does make life a bit easier by finding a possible compiler and works on different platforms (windows, osx etc). In the background, the only thing that happens is that subprocess.Popen is called with the arguments: You can poke around in the cpython/Lib/distutils/ccompiler.py and see what it does. And on line #910 is where the compiler and linker code gets executed. Once the source code in c_code is compiled, we can call compiler.link_shared_object() to create a shared object file (library). Then we can call ctypes.CDLL(lib_path) on the .so and it will behave more or less exactly as any other C-library. For instance, since we defined a main() function in our embedded source code: After we've imported the library, we can call c_lib.main() to access the function. cPython is pretty fast as is, but this allows you to create anything you want that isn't already included in the cPython core. Perhaps secure string management? Custom socket layer? Accessing low level device API's etc. I know that numpy for sure uses it quite a bit, with some modifications. Most commonly this method is probably used during setup, like pandas is doing to compile different extensions. So a more useful example of what this could be used for, would be to enhance functionality. Lets say we want to create a more accurate round() function. You could solve it by doing: But that's not the point of this exercise, so we'll use a C library for this: This lets the C code handle the number and round it to the closest matching number. The conversion back from a c_float will leave a bit of a trailing residue. So we could clean it up a bit, make it more generic for future calls and have it clean up any build files after itself even if the build crashes: It might look like a lot, but it's essentially just a with context class that cleans up any build files if it crashes. We also created a round_ function that wraps the built-in round but with better accuracy. Big thanks to Square789 over at https://pythondiscord.com/ for bouncing ideas and pointing me in the right direction when it came to the C API and ctypes. And remember if do want to use PyObject and other Python C API calls, don't forget to include /usr/include/python{sys.version_info.major}.{sys.version_info.minor}/ in the include_dirs list. As it's not automatically detected. Or your platform equivalent. Useful references on the topic: * https://docs.python.org/3/distutils/apiref.html#distutils.ccompiler.CCompiler.compile * https://docs.python.org/3/distutils/apiref.html#distutils.ccompiler.CCompiler.link_executable * https://docs.python.org/3/library/ctypes.html#return-types

I honestly have no idea where you would ever use this. But in the spirit of "why not?". Lets say you have a classic setup just like in the article Reloading Source Code in Runtime. And you want to share a variable across your application. Some settings perhaps. Sub-module also imports the config: The config is just a one thing variable: Which produces a shocking output of: Performance and risk assessment aside, there is another way you could share variables throughout your application. This is normally reserved(?) for the core of Python, but we can benefit from the fact that the core of python are as all other things, just objects. Instead of having a testconfig.py file, that everything imports, you could do (in your main application): And in your submodule.py change to the following: Isn't that kinda neat? A truly global/builtin variable called father. No need to import a file to access it. Which means you can actually change (sorta) the inner workings of Python, from Python, while running your source code. There's so many things to break, so many things to try! As a very silly example (the theme of this whole blog), we'll take the Python's builtin round() function, which quotes: What if we don't like this? We can use numpy.round_ instead. But we're obviously to lazy to correct the rest of our hypothetical application to adopt this change. So we'll patch the built-ins! Because in Python, there's no rules! Except other peoples opinions and what the core devs decide your fate should be like.. So we can patch the builtin round() by replacing it as follows: The result will be the following: Same call, just different call stacks. That to me, is amazing sorcery that should be a time honored tradition! Python allows you to shoot yourself in the foot and you're an adult, so you can take it! Even the slightest mistakes tho, like placing the import of numpy after you've modified round() will cause things to break. Not much changed, just the order of imports. But for some reason numpy doesn't like if round() was changed before the import of numpy. And you'll end up with something similar to: And this, is not something you report upstream. But it thought you a lesson didn't it? To shoot first, ask questions later!

Lets start off by setting the scene. It was a cold Swedish winter night, and you need to reload a sub-module in runtime.. You've got a pretty much standard project with a main file, a sub-module and a configuration file. The main file simply prints a value, randomizes it and prints it again. And the code might look something like this: The submodule is not much more advanced: Finally, our example configuration as follows: Which produces a shocking output of: This is arguably the most common and useful way to share data between modules in a synced manner. But what if you want to load this dynamically, in say a web server, where the sub-module (and subsequently) the config is re-imported several times per second? We can of course use importlib.reload() to achieve this, by modifying the code slightly. We'll emulate 10K imports/reloads and to time it we'll throw a simpler timer into the mix so we can see the execution time. Which on the test machine in this case will land on 2.319 seconds for a full run. If this was a web server, that would means we can support roughly 4300 users per second not counting the rest of the webserver code. This is still a bit to slow if we want to compete against the likes of lighttpd or nginx. (disclaimer: don't write your own web server, use one of the two above or whatever is hip when you're reading this) So how can we improve source code loading from here? Well, we could skip the fancy helpers and go straight to the mother dough, the importlib.util magic. Note that going "low", means there's a high probability that this will break in any minor or major version upgrade. The developers tend to not support the little guy that does ground breaking development (ironic), so use the wrappers in production! Anyway, optimizing the code: Here, we tap in to importlib.util.spec_from_file() to load the source code straight, no automagic protecting us. And if we run this snippet, we'll average around 1.734 seconds per import/reload. Not bad, that's 5700 users per second give or take depending on a few 56k modem users draging out our CPU cycles if we're not threaded. Would you ever need this? Probably not. Would it work for previous to 3.9? Maybe, probably not. Stick to importlib.reload() if you don't want things to break unexpectedly. But I'll end on another fun note. Since you have control over the namespace in which you import the module, you can change the bavior of: Since this is dependant on the namespace of the module during import, modifying importlib.util.spec_from_file_location("submodule", "./submodule.py") to something like importlib.util.spec_from_file_location("submodule.py", "./submodule.py") means you can import the module and avoid the if __name__ check to trigger. You could also use this to load source code from a remote HTTP source, not that that's ever a good idea. But you could. But you shouldn't! It does support absolute paths, which is pretty neat. It means you can locate and force imports based on paths instead of whatever the sys.path says. Global variables and local variables however is a different story, one we'll skip for this already long story.

Given the following code: This simply creates a class that supports the with context. It achieves this by having the __enter__ for when we enter the with block. But also __exit__ for when we exit the with block. The neat thing about the __exit__ function is that it mostly takes care of any exceptions for us. Exceptions will be passed as an argument to __exit__, and they can be muted by returning True from the __exit__ function. There is a caveat tho, and we can see it by running the following code: And compare it against a traditional exception handling using similar code just without the context management: We will see a result where the context management looses by quite a lot. At least if speed is of importance. It's a factor of roughly 1,42. Exceptions are pretty taxing and there's no way around it (yet), but if we ommit those. The losses of context management becomes even bigger, a factor of 2,16. This was of course performed on code executing 100M times in succession. Ideally Context Managers would only be opened once since it's the opening/closing that takes more of the execution cycles. But this might be worth keeping in mind when doing intensive operations.