Python4DataScience

Git Internals So far, we have looked at how you can use Git to manage the different states of your code. Now we want to show you the data and storage models that underlie Git. Data Model: You will be able to use...

Git 2.53 provides faster insights into the repository structure with 'git repo structure'. However, to better understand this, it is helpful to be more familiar with the Git data and storage models: www.python4data.science/en/latest/pr...
#Git

Performance Python can be used to write and test code quickly because it is an interpreted language that types dynamically. However, these are also the reasons it is slow when performing simple tasks repeatedl...

The section on performance measurements and finding bottlenecks has been significantly expanded to include cProfile/profiling.tracing, tprof, and profiling.sampling/Tachyon: www.python4data.science/en/latest/pe...
#Python #Performance

Document In order for your software package to be useful, documentation is required that describes how your software can be installed, operated, used and improved: Those who want to use your package need in...

We have updated the documentation section with references to README, CONTRIBUTING, CHANGELOG, etc.
python-basics-tutorial.readthedocs.io/en/latest/do...
#Python #Documentation

Extensions Administration: SQLAlchemy Admin for Starlette/FastAPI, Flexible admin interface for SQLAlchemy models. Downloads Contributors Commit activity Licence,, Piccolo Admin, Simple but powerful admin int...

We have updated the FastAPI extensions. It is very surprising to us that millions of extensions are being downloaded that have not been updated for over a year.
www.python4data.science/en/latest/da...
@fastapi.tiangolo.com
#Python #FastAPI #REST

Unicode and character encodings Special characters and escape sequences:\n stands for the newline character and\t for the tab character. Character sequences that begin with a backslash and are used to represent other characters a...

I took a look at the changes coming with Python 3.15 – and I can't wait to put them to productive use. I've already updated our tutorials:
• Performance measurements: www.python4data.science/en/latest/pe...
• Tachyon: www.python4data.science/en/latest/pe...
#Python

pytest pytest is an alternative to Python’s Unittest module that simplifies testing even further. pytest automatically recognises tests based on filenames and functions that start with test_, while unitte...

We have updated the section on pytest with many exciting use cases
* on command line options
* on generating markers
* and on parameterising exceptions
python-basics-tutorial.readthedocs.io/en/latest/te...
#Python #Testing #pytest

Precision-Recall-Curve comparison between workspace and HEAD

Receiver operating characteristic (ROC) comparison between workspace and HEAD

Confusion Matrix comparison between workspace and HEAD

We have updated our tutorial to data management with DVC. It also allows you to create lightweight data science and data modelling workflows and execute them in a parameterised manner: www.python4data.science/en/latest/pr...
#Data #Versioncontrol #Git #DataScience #Modeling #Python

Configuring Claude Code or Cursor for uv How do we configure Claude Code or Cursor to automatically use uv instead of pip for Python package management? Claude Code Claude Code uses CLAUDE.md files to configure your project’s storage and ...

Now we have also described how to use uv reliably for Cursor: www.python4data.science/en/latest/pr...
#CursorAI #Python #Packaging #uv

Configuring Claude Code for uv How do we configure Claude Code to automatically use uv instead of pip for Python package management? Claude Code uses CLAUDE.md files to configure your project’s storage and context, ensuring a co...

We have now described how to create a configuration for Claude Code so that it uses uv reliably: python4data.science/en/latest/pr...
#ClaudeCode #Python #Packaging #uv

pandas pandas is a Python library for data analysis that has become very popular in recent years. On the website, pandas is described thus: „pandas is a fast, powerful, flexible and easy to use open sourc...

Since we have recently been asked frequently whether pandas is slow and whether we should use Polars, Dask or DuckDB instead, we have now provided an initial overview of the various technologies: www.python4data.science/en/latest/wo...
#Python #Performance #DuckDB

Files and directories pathlib implements path operations using pathlib.PurePath and pathlib.Path objects. The os and os.path modules, on the other hand, offer functions that work at a low level with str- and bytes which...

We have now completely switched to pathlib: python-basics-tutorial.readthedocs.io/en/latest/sa...
#Python

Ruff Ruff is an extremely fast Python linter and code formatter written in Rust that can enforce the rules of flake8, isort, perflint, Black, Bandit, and others. In total, Ruff can check over 800 rules....

We have finally documented Ruff – the tool greatly simplifies static code analysis for Python projects: www.python4data.science/en/latest/pr...
#Python #Ruff

Creating a distribution package Distribution Packages are archives that can be uploaded to a package index such as pypi.org and installed with pip. Structure: A minimal distribution package can look like this, for example: pyproj...

We have now updated our packaging tutorial to include PEP 639, which enables SPDX-compliant licensing: python-basics-tutorial.readthedocs.io/en/latest/pa...
#Python #Packaging #SPDX #Licensing

JSON Overview:,,, Data structure support,+-, JSON supports array and map or object structures and many different data types including strings, numbers, boolean, null etc., but no date formats. However, ...

We have added a section on additional JSON tools: www.python4data.science/en/latest/da...
#Python #JSON

Geodata File formats: PMTiles: PMTiles is a general format for tile data addressed by Z/X/Y coordinates. This can be cartographic vector tiles, remote sensing data, JPEG images or similar. HTTP Range Reque...

We have added several geopython libraries: www.python4data.science/en/latest/da...
#Python #Geospatial #GeoPython

Licensing In order for others to use your software, it should have one or more licences that describe the terms of use. Otherwise, it is likely to be protected by copyright. Authors are those who have origin...

We have significantly expanded the section on licences for AI systems: www.python4data.science/en/latest/pr...
#AI #Licensing #OpenData #OpenSource

💥Spack v1.0 is out!💥

This is a huge milestone. We reworked the core to add compiler dependencies, and we're introducing a stable package API.

🚀1.0 also adds concurrent builds, better includes, and much more -- read it all in the release notes!

github.com/spack/spack/...

XKCD #3117: Replication Crisis

The XKCD comic on reproducible scientific results fits perfectly with our tutorial 🧐 😉
www.python4data.science/en/latest/pr...

Graph from GitHub’s Octoverse 2024 report showing a spike in utilization of Jupyter Notebooks across GitHub. This is calculated by looking at the distinct number of public repositories with at least one Jupyter Notebook by the year the repository was created. Since 2016, we have seen this number surge from near zero to more than 1.5 million repositories using Jupyter Notebooks.

Almost more significant than the success of #Python is the growth of #Jupyter #Notebooks: “Data scientists and machine learning researchers commonly use the #OpenSource application for #MachineLearning, #DataViz, and more.”
jupyter-tutorial.readthedocs.io/en/latest/in...

Protomaps Protomaps is an open source project for the creation and use of vector maps. It was developed as a lightweight alternative to conventional map providers and offers a number of advantages. Open Sour...

We have added a section on protomaps to our PyViz tutorial. Protomaps makes map visualisations so much easier.
pyviz-tutorial.readthedocs.io/en/latest/pr...
#Protomaps #Geography #World #Map @protomaps.com

Geodata File formats: PMTiles: PMTiles is a general format for tile data addressed by Z/X/Y coordinates. This can be cartographic vector tiles, remote sensing data, JPEG images or similar. HTTP Range Reque...

We have expanded the section on geodata to include the most common (tile) file formats: www.python4data.science/en/latest/da...
#Geography #GIS

Licensing In order for others to use your software, it should have one or more licences that describe the terms of use. Otherwise, it is likely to be protected by copyright. Authors are those who have origin...

And a new section on AI/ML licences has also been added: www.python4data.science/en/latest/pr...
#AI #ML #License

Docstrings With the Sphinx extension sphinx.ext.autodoc, docstrings can also be included in the documentation. The following directives can be specified … for function-like objects: … for data and attributes:...

We have updated our Python Basics tutorial to describe the guidelines for docstrings in more detail:
python-basics-tutorial.readthedocs.io/en/latest/do...
#Python #Documentation #DX

Licensing In order for others to use your software, it should have one or more licences that describe the terms of use. Otherwise, it is likely to be protected by copyright. Authors are those who have origin...

We have expanded the section on open source hardware licences to include the TAPR and Solderpad Hardware Licence: www.python4data.science/en/latest/pr...
#OpenSource #Hardware #Licence

Technology Radar | Guide to technology landscape The Technology Radar is an opinionated guide to today's technology landscape. Read the latest here.

thoughtworks Technology Radar has now also adopted the tools uv and Renovate: www.thoughtworks.com/radar

GitLab CI/CD GitLab CI/CD can automatically build, test, deploy and monitor your applications during iterative code changes. This reduces the risk that you will develop new code based on buggy previous versions...

We have expanded our section on GitLab CI/CD pipelines with examples of
• GitLab Pages
• npm deployments with rsync
• building Docker containers
• multi-arch images with Buildah
• migrating GitHub Actions
www.python4data.science/en/latest/pr...
#GitLab #CICD #DevOps #DX

How LLMs help us with programming We were recently asked by a global chemical company if we could give their engineers an introduction to programming with Python and Large Language Models (LLM). Their expectations of what they wanted ...

We have written down our experiences of how LLMs help us with programming: cusy.io/en/blog/how-...
#LLM #AI #programming #DX #Python

Voilà vs. Panel A major difference between Panel and Voilà lies in the processing of the notebooks: Voilà is based directly on the notebook format and transfers the entire output to the Voilà dashboard, whereas in...

Which Python dashboard library for which purpose?
We were left with only two candidates: Voilà and Panel: jupyter-tutorial.readthedocs.io/en/latest/da...
#DataViz #Python

🎉 4000 Pythonistas and data scientists now follow us on Bluesky 🤗 We are very pleased about the great interest in our offer.
#Python #DataScience

Our course for the versioned and reproducible storage of code and data in data science workflows is now also referenced in the official Git documentation: git-scm.com/doc/ext
#Git #DataScience #DX