CS - Python, Jupyter, and alike

Python
Jupyter
rise.JS
A personal cheat sheet of Python and Jupyter tips, tools and workflows.
  1. Tooling
  2. Tool

Personal notes on Python and Jupyter — installing packages the right way, teaching with notebooks, handling PDFs, data-visualization resources and a few recurring fixes.

Jupyter & JupyterLab

Install packages safely from a notebook

Inside a notebook, !pip install and !conda install run in the shell, whose environment can differ from the kernel’s Python — so packages may end up installed where the running kernel can’t see them. The safe pattern ties the install to the current kernel’s prefix (Anaconda assumed):

import sys
!conda install --yes --prefix {sys.prefix} <package>

For pip, the equivalent is !{sys.executable} -m pip install <package>. See Jake VanderPlas’s write-up for the full discussion.

Enable the Python debugger

Use the xeus-python kernel (in the base env or a fresh conda env) together with the debugger extension:

conda install xeus-python notebook -c conda-forge
jupyter labextension install @jupyterlab/debugger

Since JupyterLab 3.0 the visual debugger ships built in, so the labextension install step is only needed on older JupyterLab versions. See the debugger and xeus-python repositories.

Choose the browser Jupyter opens

To open Jupyter in a browser other than the system default, generate a config and edit c.ServerApp.browser in $HOME/.jupyter/jupyter_server_config.py:

jupyter server --generate-config   # or: jupyter notebook --generate-config
# Default: ''
c.ServerApp.browser = u'/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome %s'

The webbrowser module lists the browsers you can register. More detail in this discussion.

Lecturing with RISE

RISE turns a notebook into a reveal.js slideshow.

Slideshow troubleshooting

RISE once stopped working in Safari after updating to macOS Big Sur 11.2.3. Steps that fixed it:

  • With RISE 5.7.1 there was a dependency clash:
pkg_resources.ContextualVersionConflict:
(jedi 0.15.2 ... Requirement.parse('jedi>=0.16'), {'ipython'})
  • conda update jedi would not bump jedi to 0.18 (available on PyPI). It only worked with conda update -c conda-forge jedi=0.17.2, after a long “frozen/flexible solve… solving environment…” wait.
  • After that, Cell Toolbar > Slideshow was visible in Safari, but the button to launch the RISE slideshow never appeared.
    • Current workaround: use Chrome (see Choose the browser Jupyter opens).
    • If it persists, run jupyter-nbextension install rise --py --sys-prefix and the button should (luckily) show up.

Customizing the chalkboard and themes

"rise": {
  "chalkboard": {
    "boardmarkerWidth": 3,
    "grid": {
      "color": "rgb(50,50,10,0.5)",
      "distance": 80,
      "width": 2
    }
  }
}

Teaching, grading & publishing

Learning resources

Resource Description
Teaching and Learning with Jupyter Reference e-book for anyone using Jupyter for lecturing.
Lorena Barba’s group Open-source code-sharing initiative with excellent Jupyter notebooks for CFD.

Journals

Journal Description
Journal of Open Source Software (JOSS) Open-source-focused journal; many machine-learning, data-science and Jupyter-based projects, with renowned people involved.

Autograding tools

Tools for testing Jupyter notebooks and managing assignments:

Tool Description
nbgrader Autograding and assignment management, by the Jupyter team.
autograde Autograding tool from CSSH, RWTH Aachen.
Otter-Grader Autograder from UC Berkeley.

See also: a report comparing graders and the alternative Web-CAT; a proposed fix for an alleged nbgrader security issue by Aalto’s SciComp group; and a video on using a grader with GitHub Classroom, with its handout.

Working with PDFs

PDF parsing & editing libraries

Library Description
pdfminer.six Text extraction; pip install pdfminer.six, then run pdf2txt.py file.pdf (added to PATH).
pypdf / PyPDF2 General-purpose read/write. Note: PyPDF2 is deprecated — use the maintained pypdf.
WeasyPrint Convert HTML/CSS to PDF.
ReportLab Programmatic PDF generation.
pyFPDF (fpdf2) PDF generation; port of PHP’s FPDF.
Pyppeteer Python port of Puppeteer for headless Chrome/Chromium automation.
PDFKit JavaScript PDF generation for Node and the browser.

Editing the font color of a PostScript-generated PDF

An unfinished experiment (no clean solution yet). The goal was to recolor the black text of a LaTeX-generated PDF into a CMY-balanced color (after K = 0), to use up CMY printer cartridges. Two avenues to try:

  • Follow Stefano Chizzolini’s approach with PDFClown, sketched in this post — though recent PDFClown versions may already bundle those isolated Java elements.
  • Try PyPDF2 in Python; there is a tutorial, but changing font color looks hard. Left for the future.

Data visualization

Resource Description
PyViz Overview of the Python data-visualization ecosystem.
HoloViz Modern project that integrates many visualization tools.
From Data to Viz Guide to picking the right chart — a holistic, design-meets-data-science view.
Fundamentals of Data Visualization Book by Claus O. Wilke.
Interactive Dashboards and Data Apps with Plotly and Dash Book by Elias Dabbas.

Python snippets & fixes

CSV → Markdown table

Use MDTable:

from mdtable import MDTable
MDTable('input.csv').save_table('output.md')

Printing Unicode music symbols

To print musical-note symbols you need a font capable of rendering the relevant Unicode block (see this thread and the music Unicode chart). For example, the G clef uses the \U escape:

print('\U0001D120')

Few fonts handle music symbols; one option is Bravura.

Python “gotchas”

A collection of Python “gotchas” — surprising behaviors worth knowing about.

Fixing ModuleNotFoundError in VS Code’s debugger

When a virtual environment (e.g. a conda env) is involved, this is usually a pip/environment mismatch. To fix it:

  • Check that which pip points to the current environment. If not, install with the environment’s own interpreter: python -m pip install <package>, then confirm with pip list.
  • Press Cmd/Ctrl + Shift + P and switch to the Python interpreter for your environment.
  • Add "env": { "PYTHONPATH": "${workspaceRoot}" } to the relevant configuration in launch.json.
  • Apply all of the above and restart VS Code.
Back to top