In another post, I mentioned separating my computer environment from the company’s one when I bought Miorine (my M2 Max MBP). I also set up my private development environment with Neovim because I thought using my company’s JetBrains licenses on my personal laptop was inappropriate. It was interesting because it was my first time (after 30 years since I learned vi in college) readily accepting vi/vim for my main flow. And there was a small challenge yesterday when I decided to set up a Python environment.
I haven’t used Python even in my work because 1) the primary language for most of my tasks is Go, and moreover 2) a psychological barrier. I should have used Python 3.7 for work as there are some dependencies (as usual), but the problem was that the Apple Silicon environment didn’t support Python 3.7; that means I had to use Rosetta (and Rosetta iTerm terminal), which I wasn’t up to.
So, I put some distance from Python at work — anyhow, I haven’t had any tasks that required Python. However, at this moment, when everything was getting back to normal, I got my first COVID and was isolated in my bedroom to protect other family members. I needed something to motivate myself, so I started watching ML and data processing (e.g., Pandas, Numpy, TensorFlow, etc.) lectures on YouTube.
I wanted to try some sample code, and the fact that I didn’t need to be bothered by a specific Python version and Rosetta at home invigorated me. So, I decided to set up a genuine (Apple Silicon native, and based on Neovim) Python development environment on my M2 Max MBP.
Installing Python
Let’s get it straight; I don’t like Python. There would be many reasons, but the most significant one is there are too many ways to do something. Just installing M1/M2 native Python could be done in many ways. Apple has its own guide, but it produced some warnings that made me a little reluctant to use it. After some investigations, my choice was to use miniforge to install Conda and manage the Python version and environment: creating a Conda environment and using one Python version for it. Miniforge? Conda? Yes, it’s confusing. As I said, that’s why I don’t like the Python environment. We need to clarify some concepts.
- Conda is an open-source package and environment management system: a package management system such as pipand an environment management system like venv — simultaneously.
- Anaconda is a set of hundreds of packages, including Conda, Numpy, etc. The catchphrase of Anaconda is “Where packages, notebooks, projects, and environments are shared.” It is also worth to note Anaconda is a private company.
- Miniconda is a small subset of Anaconda with only Conda and its basic dependencies.
- Conda-forge is a GitHub community-driven organization containing repositories of Conda recipes. It is an alternative channel for package installations.
- Miniforge (finally!) is a GitHub repository and a community (Conda-forge) driven minimal Conda installer.
While Anaconda and Miniconda did not support the Apple Silicon environment (although now Miniconda supports arm64), Miniforge provided an installer for arm64 (Apple Silicon) architecture. That’s why I was going to use Miniforge.
Download the Miniforge3 arm64 installer script from the miniforge git repository.
![](https://cdn-images-1.medium.com/max/1024/0*V9aV4u2NgH_AjkAF.jpg)
Run the script. It will install arm64 Conda and Python 3.10.10 (the latest one).
sh ~/Downloads/Miniforge3-MacOSX-arm64.sh
Once installed, we rerun the terminal, and it will automatically activate the Conda base environment. You can install any additional packages (e.g., pandas) either using pip install or conda install.
Some guides had used Pyenv for multiple Python versions, but I thought it was unnecessary (at least for me).
Additionally, you can install TensorFlow for macOS (using Apple Silicon) and the tensorflor-metal plug-in.
pip install tensorflow-macos
pip install tensorflow-metal
Lastly, I verified installations with the following script, as noted in Apple’s guide.
import tensorflow as tf
cifar = tf.keras.datasets.cifar100
(x_train, y_train), (x_test, y_test) = cifar.load_data()
model = tf.keras.applications.ResNet50(
include_top=True,
weights=None,
input_shape=(32, 32, 3),
classes=100,)
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(optimizer="adam", loss=loss_fn, metrics=["accuracy"])
model.fit(x_train, y_train, epochs=5, batch_size=64)
Everything seemed to be perfect. However, the trouble came with the Neovim.
Neovim for Python
I had set up pyright for the Python LSP server. So, I expected it would work seamlessly. What I don’t like about the modern GUI IDE is that it usually has its own execution environment. Therefore, we have to set up a new (different from the existing terminal) environment for the IDE. If I cannot build the source code while I can build it in the terminal, that’s the typical implication of these separate environments. As Neovim (vim) runs in the terminal environment it is launched from, they share the environment. We don’t need to set up another environment for it.
It should have been easy… so I was surprised when I typed the first line of a Pandas sample code.
![](https://cdn-images-1.medium.com/max/984/0*hsXyzEnNEDLCfHcM.png)
What?… Neovim (precisely, pyright) couldn’t resolve the pandas package? I checked the installed packages with the conda list command, and it (pandas) was definitely there.
![](https://cdn-images-1.medium.com/max/1024/0*m-VGPJQBanUe8eL2.png)
At first, I guessed that pyright couldn’t resolve the current (base) Conda environment. I created a pyrightconfig.json file and tried to specify stubPath, venvPath, and so on. Nothing worked. I checked several Reddit and Stack Overflow articles. I’ve come across a few similar cases, but unfortunately, none of them were of much help. It’s frustrating to see that something should work smoothly, yet it doesn’t. This is a real source of agony for developers. I tried several other Python LSP servers, such as the jedi-language-server and the anakin-language-server (what???). However, it didn’t make any difference. Wait? No difference at all?
After all this, I tried pyright via the command line. It showed clean results. pyright had worked.
![](https://cdn-images-1.medium.com/max/1024/1*6aoC3_Bap7OJd5RT1qz75Q.png)
I checked the current file’s LSP status with the LspInfo command.
![](https://cdn-images-1.medium.com/max/1024/0*psT8i_ChMCcUSnXJ.png)
Ah-ha. pyright wasn’t the only one involved in the diagnostics. Null-ls was another player behind the scene. It should have come along with a soullessly copied config file. I disabled Python diagnostics from null-ls.
![](https://cdn-images-1.medium.com/max/1024/0*H13JR-xggDDjUTdC.png)
Bingo. The error was gone — piece of mind.
![](https://cdn-images-1.medium.com/max/1024/0*MyW4eIcjpWaH1_aO.png)
If I had only copied configs and used features without understanding the internals, I couldn’t resolve it and should get back to PyCharm. As an engineer, finding solutions by thoroughly comprehending the underlying systems is a source of immense enjoyment. Truly it is.
Using Python on M2 was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.