Consortium for Python Data API Standards

Machine Learning is becoming critical infrastructure for many organizations. At the same time, the current tensor and dataframe infrastructure at the foundation of machine learning is fragmented. This makes it difficult for a robust ecosystem to develop around the state of the art.

Even as Python has become a common language to express workflows, the foundational data-structures used in Python such as n-dimensional arrays (tensors) and dataframes have become more divided. For example, tensors are fragmented between TensorFlow, PyTorch, NumPy, CuPy, MXNet, JAX, Xarray, Dask, and others. Dataframes are fragmented between Pandas, cuDF, Vaex, Modin, Dask, Koalas, PySpark, Turi Create, Apache Arrow, and more.

The differences in APIs between these projects provide marginal benefit but also cause inconsistencies and incompatibilities that require additional effort, limit participation, prevent re-use, and slow progress. Innovation and differentiation are most valuable when created using standard building blocks that ensure consistency and compatibility.

Quansight Labs is forming a consortium of leading companies to construct an API standard for both tensors and dataframes, which are the critical building blocks for machine-learning workflows. The standard will be coordinated with the wider open-source community in its creation and later dissemination. Consortium members are able to participate early in the process of defining the standard. The early requirements-gathering phase leading to the first draft of the standard is being conducted with consortium members including key community participants. This drafting phase is followed by public dissemination, discussion, and refinement of the standard within the NumPy, Pandas and wider PyData communities.

Quansight Labs is well positioned to establish this Consortium because of its expertise with the technologies involved, understanding of the open-source process, strong relationships with many of the relevant projects, and because Quansight Labs does not have commercial interest in the particulars of the standard.

We are looking for around 10 member companies to join the Consortium. Members will be able to participate in consortium meetings and decision making. The funding obligation to participate is $50,000 per member for the first year in order to support the development of the standards.

Definition of the API standards has commenced, with the first RFCs having been published, and the consortium is taking shape - see data-apis.org. Organizations already committed to the consortium include Quansight, Intel, Microsoft, Google Research and DE Shaw.