Following system colour scheme Selected dark colour scheme Selected light colour scheme

Python Enhancement Proposals

PEP 9999 – JSON Package Metadata

Author:
Emma Harper Smith <emma at python.org>
PEP-Delegate:
Paul Moore
Discussions-To:
Pending
Status:
Draft
Type:
Standards Track
Topic:
Packaging
Created:
09-Dec-2025
Post-History:
Pending
Table of Contents

Abstract

Python package metadata (“core metadata”) was first defined in PEP 241 to use RFC 822 email headers to encode information about packages. This was reasonable at the time; email messages were the only widely used, standardized text format that had a parser in the standard library at the time. However, issues with handling different encodings, differing handling of line breaks, and other differences between implementations have caused numerous packaging bugs. To resolve these issues, this PEP proposes introducing a Javascript Object Notation (JSON) encoded file containing core metadata in Python packages.

Motivation

The email message format has a number of complexities and limitations which reduce its utility as a portable textual interchange format for packaging metadata. Due to the email parser requiring configuration changes to properly generate valid core metadata, many projects do not use the email module and instead generate core metadata in a custom manner. There are many pitfalls with generating email headers that these custom generators can hit. First, core metadata fields may contain newlines in the value of fields. These newlines must be handled properly to “unfolded” multiple lines per RFC 822. Improperly escaped newlines can lead to generating invalid core metadata. Second, as discussed in the core metadata specifications:

The standard file format for metadata (including in wheels and installed projects) is based on the format of email headers. However, email formats have been revised several times, and exactly which email RFC applies to packaging metadata is not specified. In the absence of a precise definition, the practical standard is set by what the standard library email.parser module can parse using the email.policy.compat32 policy.

Since no specific email RFC is selected, the current core metadata specification is ambiguous whether a given core metadata document is valid. RFC 822 is the only email standard to be explicitly listed in a PEP. However, the core metadata specifications also requires that core metadata is encoded using UTF-8 when written to a file. This de-facto makes the core metadata follow RFC 6532, which specifies internationalization of email headers. This has practical interoperability concerns. Until a few years ago, it was unspecified how to handle non-ASCII encoded content in core metadata, causing confusion about how to properly encode non-ASCII emails in core metadata. Third, the current format is difficult to properly validate and parse. Many tools do not check for issues with the output of the email parser. If a document is malformed, it may still parse without error by the email module as a valid email message. Furthermore, due to limitations in the email format, fields like Project-Url must create custom encodings of nested key-value items, further complicating parsing. Finally, the lack of a schema makes it difficult to validate the contents of email message encoded metadata. While introducing a specification for the current format has been discussed previously, no progress had been made, and converting to JSON was a suggested resolution to the issues raised.

Rationale

Introducing a new core metadata file with a well-specified format will greatly ease generating, parsing, and validating metadata. JSON is a natural choice for storing package core metadata. It is easily machine readable and writable, is understandable to humans, and is well supported across many languages. Furthermore, PEP 566 already specifies a canonicalization of email formatted core metadata to JSON. JSON is also a frequently used format for data interchange on the web. For discussion of other formats considered, please refer to the rejected ideas section.

To maintain backwards compatibility, the JSON metadata file MUST be generated alongside the existing email formatted metadata file. This ensures that tools that do not support the new format can still read package metadata for new packages.

The JSON formatted metadata file must be semantically equivalent to the email encoded file. This ensures that the metadata is unambiguous between the two formats, and tools may read either when both are present. To maintain performance, this equivalence is not required to be verified by installers, though other tools may do so. Some tools may choose to make the check dependent on a configuration flag.

Package indexes SHOULD check that the metadata files are semantically equivalent when the package is added to the index. This is a low-cost, one-time check that ensures users of the index are served valid packages.

Specification

JSON Format Core Metadata File

A new optional file METADATA.json shall be introduced as a metadata file for Python packages. If generated, the METADATA.json file MUST be placed in the same directory as the current email formatted METADATA or PKG-INFO file.

For wheels, this means that METADATA.json MUST be located in the .dist-info directory. The wheel format minor version will be incremented to indicate the change in the format.

For source distribution packages, the METADATA.json file MUST be located in the root directory of the project sources. Tools that prefer the JSON formatted metadata file MUST check for the existence of a METADATA.json in the source distribution before reading the file.

The semantic contents of the METADATA and METADATA.json files MUST be equivalent if METADATA.json is present. Installers MAY verify this information. Public package indexes SHOULD verify the files are semantically equivalent.

Conversion to JSON Encoding

Conversion from the current email format for core metadata to JSON should follow the process described in PEP 566, with the following modification: the Project-URL entries should be converted into an object with keys containing the labels and values containing the URLs from the original email value. The overall process thus becomes:

  1. The original key-value format should be read with email.parser.HeaderParser;
  2. All transformed keys should be reduced to lower case. Hyphens should be replaced with underscores, but otherwise should retain all other characters;
  3. The transformed value for any field marked with “(Multiple-use”) should be a single list containing all the original values for the given key;
  4. The Keywords field should be converted to a list by splitting the original value on commas;
  5. The Project-URL field should be converted into a JSON object with keys containing the labels and values containing the URLs from the original email value.
  6. The message body, if present, should be set to the value of the description key.
  7. The result should be stored as a string-keyed dictionary.

One edge case in the above conversion is that the Project-URL label is “free text, with a maximum length of 32 characters.” This presents a problem when trying to decode the label. Therefore this PEP sets the requirement that the Project-URL label be any text except the comma (,) character. This allows for unambiguous parsing of the Project-URL entries by splitting the text on the left-most comma (,) character.

JSON Schema for Core Metadata

To enable verification of JSON encoded core metadata, a JSON schema for core metadata has been produced. This schema will be updated with each revision to the core metadata specification. The schema is available in Appendix: JSON Schema for Core Metadata.

TODO: where should the schema be served/what should the $id be?

Serving METADATA.json in the Simple Repository API

PEP 658 introduced a means of serving package metadata in the Simple Repository API. The JSON encoded version of the package metadata may also be served, via the following modifications to the Simple Repository API:

A new attribute data-dist-info-metadata-json may be added to anchor tags in the Simple API. This attribute should have a value containing the hash information for the METADATA.json file in the same format as data-dist-info-metadata. If data-dist-info-metadata-json is present, the repository MUST serve the JSON encoded metadata file at the distribution’s path with .metadata.json appended to it. For example, if a distribution is served at /simple/foo-1.0-py3-none-any.whl, the JSON encoded core metadata file MUST be served at /simple/foo-1.0-py3-none-any.whl.metadata.json.

Deprecation of the METADATA and PKG-INFO Files

The METADATA and PKG-INFO files are now deprecated. This means that a future PEP may make the METADATA and PKG-INFO files optional and require METADATA.json to be present. Please see the next section for caveats to that change.

Despite the METADATA and PKG-INFO files being deprecated, new core metadata revisions should be implemented for both JSON and email to ensure that they may remain semantically equivalent.

Backwards Compatibility

The specification for METADATA.json is designed such that the new format is completely backwards compatible. Existing tools may read metadata from the existing email formatted files, and new tools may take advantage of the new format.

A future major revision of the wheel specification may make the METADATA and PKG-INFO files optional and make the METADATA.json file required. Note that tools will need to maintain parsing of email metadata indefinitely to support parsing metadata for old packages which only have the METADATA or PKG-INFO files.

Security Implications

One attack vector with JSON encoded core metadata is if the JSON payload is designed to consume excessive memory or CPU resources in a denial of service attack. While this attack is not likely to affect users whom can cancel resource-intensive operations, it may be an issue for package indexes.

There are several mitigations that can be made to prevent this:

  1. The length of the JSON payload can be restricted to a reasonable size.
  2. The reader may use a JSONDecoder to omit parsing int and float values to avoid quadratic number parsing time complexity attacks.
  3. I plan to contribute a change to the JSONDecoder in Python 3.15+ that will allow it to be configured to restrict the nesting of JSON payloads to a reasonable depth.

With these mitigations in place, concerns about denial of service attacks with JSON encoded core metadata are minimal.

Reference Implementation

A reference implementation of the JSON schema for JSON core metadata is available in Appendix: JSON Schema for Core Metadata.

Furthermore, a reference implementation in the packaging library is available.

Rejected Ideas

Using Another File Format (TOML, YAML, etc.)

While TOML or another format could be used for the new core metadata file format, JSON has been chosen for a few reasons:

  1. Core metadata is mostly meant as a machine interchange format to be used by tools and services which wish to interoperate. Therefore the human-readability of TOML is not an important consideration in this selection.
  2. JSON parsers are implemented in many languages’ standard libraries and the json module has been part of Python’s standard library for a very long time.
  3. JSON is fast to parse and emit.
  4. JSON schemas are JSON native and commonly used.

Open Issues

Where Should the JSON Schema be Served?

Where should the standard JSON Schema be served? Some options would be packaging.python.org, pypi.org, python.org, or pypa.org.

My first choice would be packaging.python.org, but I am open to other options.

Should we also update the WHEEL metadata file format to be JSON encoded?

The WHEEL metadata file format is also an email formatted file. This means that it is subject to the same parsing and validation issues as the METADATA and PKG-INFO files. However, the WHEEL file is part of the initial wheel format version check done by installers. Changing the file format might harm backwards compatibility by making old installers unable to read new metadata.

I think it could make sense to introduce a WHEEL.json file. Then a future wheel major version could remove the WHEEL file and require the WHEEL.json file instead.

Source: https://github.com/python/peps/blob/main/peps/pep-9999.rst

Last modified: 2025-12-09 21:50:33 GMT