Boost Histogram for Python - Developing - Lovely Buggies

Once you’ve mastered boost-hist, it’s time to consider becoming a developer to contribute to Scikit-HEP! It’s a creative and challenging job. So let’s see how to build a developing environment and become a developer!

Building from Source

The first thing we need to do is to clone the source code from the official git.

git clone --recursive https://github.com/scikit-hep/boost-histogram.git
cd boost-histogram

Pip Environment

Then let’s build our developer environment!

In this part, we need to establish a virtual environment to develop in a good manner. This is how you would set one up with Python 3:

Create a python 3 environment named .env and activate it.
```
python3 -m venv .env
source .env/bin/activate
```

Install modules needed from PyPI with pip3.

pip3 install numpy ipykernel pytest-sugar numba matplotlib

Create a kernel named boost-hist.

python3 -m ipykernel install --user --name boost-hist

Specify pip extra requirement - VCS projects can be installed in editable mode (using the –editable option) or not. You can install local projects or VCS projects in “editable” mode:
```
pip3 install -e .[test]
```
Deactivate .env.
```
deactivate
```

Now, you can run notebooks using your system Jupyter Lab (the next generation Jupyter Notebook), and it will list the environment as available (view your kernel according to jupyter kernelspec list)!

jupyter lab

To rebuild, you may need to delete the /build directory and rerun pip3 install -e . from the environment.

rm -r ./build
pip3 install -e .

You can also build your developer environment using CMake. I tried it and found it not convenient as pip3.

Testing

Now we can test our project using pytest.

Just re-activate our environment and run pytest is ok.

P.S. As a developer, I think you should also have pytest installed in your pip root sake of convenience. Of course, this is NOT a good developing manner, but sometimes convenience and manner is a tradeoff.

python3 -m pytest tests

When you see the green 100% progress bar, the testing is completed.

If you want to benchmark before and after a change, you can use the following commands:

python3 -m pytest tests --benchmark-enable --benchmark-autosave

# Make some changes

python3 -m pytest tests --benchmark-enable --benchmark-autosave

pip3 install pygal pygaljs

pytest-benchmark compare 0001 0002 --sort fullname --histogram

Let’s see what’s shown by adding a benchmark to our command.

For each pytest, there are several tests in benchmarks. The best performance concerning each criteria for a unit test is shown in green, while the worst is in red. To compare the testings, we can run pytest-benchmark again after some modifications (here I made no changes to the original tests).

Then, we can compare the two testings to see the influence of the modifications. (Note, while the histogram option (--histogram) is nice, it does require pygal and pygaljs to be installed. Feel free to leave it off if not needed.) Except for the testing display, the benchmarks will show the comparison results by exporting svg figures.

Q&A

Q1: CMake was unable to find the program ‘Ninja’.

A1 (click to expand)

When building my environment, I met an error CMake Error: CMake was unable to find a build program corresponding to 'Ninja'. I browsed some solutions, but only to find that nothing changed.

I guessed maybe ninja is not installed in local path usr/bin and tried to move ninja binary to it. But system did not allow for that operation.
I also tried to symlink "ninja-build" to "ninja" according to ln -s /usr/bin/ninja /usr/bin/ninja-build OR ln -s /usr/local/bin/ninja /usr/local/bin/ninja-build, but the error still existed.

If your situation is same as mine and you cannot solve this error by using the ways mention above, I recommend you to use the powerful pytest for your unit test.

Q2: Boost-hist already exist in … when rebuilding.

A2 (click to expand)

When trying to rebuild in a different director by pip3 install -e ., we will meet the problem: boost-hist already exist. The simplest solution is to move your project folder to the right place. You can see the existed installation in the last line of error thrown.

Q3: Pytest lacks ‘pybind11_tests’.

A3 (click to expand)

When we are testing using python3 -m pytest, a normal error is ModuleNotFoundError: No module named 'pybind11_tests'. I reported this bug and proposed an issue #312, Henry gave me the solution timely:

I guess I habitually run python3 -m pytest tests, which forces the tests dir to be the only place searched. Without that, we aren't limiting the search locations, so it picks up extern/pybind11/tests, which it (obviously) should not pick up. For now, you can add the tests part to your command, and we can add a pytest configuration option to disable searching for tests in extern. I can help add that soon.

Q4: No modules named ‘pytest’ found.

A4 (click to expand)

If you are puzzled by this issue, you might pip install pytest in your virtual environment. Of course, pip root cannot find your pytest after your deactivation. Note that you are still in original folder if you activate your virtual environment (ls -a can see), so just run python3 -m pytest tests is ok, instead of python3 - m pytest ../tests, else you will still meet lacks "pybind11_tests" problem, i.e. Q3.

Contribution

Questions: #318.
Ideas: #311.
Report Bugs: #312 (fixed by Henry #313), #319.
Fix Bugs: #315 (merged), #317 (merged).

Conclusion

Testing is a slow process, keep patient and enjoy your time as a developer!

Boost Histogram for Python - Developing