11  Release process

A release is a snapshot of the data package at a specific point in time. It contains the data and metadata in their final state at that point in time and is tagged with a specific version number. Releases are how we track changes over time and make it easier to share or distribute data packages to researchers in a structured and predictable way.

The release process is dependent on the type of data contained in the data package. For data packages that contain human (in particular health or sensitive) data, the release process is a bit more complicated, as will be explained later, than for data packages that contain data that doesn’t fall under legal restrictions (e.g. GDPR). This is because the data must remain on secure servers and can’t (usually) be uploaded to any public repository or archive. That means we can’t use services like GitHub workflows nor upload any data for public access, so we can’t use a continuous release process (or at least it is more difficult to).

For nonsensitive data, the release process is fairly simple. It can be done through a GitHub workflow (e.g. release.yml) that runs on a schedule or is triggered by specific events (e.g. a specific type of commit, described later in this chapter). The data and metadata can be attached as release artifacts on GitHub and/or be uploaded to public archives like Zenodo.

Regardless of the type of data, the general steps remain mostly the same with some notable differences. The difference is in where the release is done (the computing environment where the release process is executed, e.g. GitHub or on a secure server) and what triggers a release. We’ll start with the triggers.

11.1 Triggers

There are a few ways to trigger the release process: manually, on a schedule, or based on specific events whenever a change is merged into main. While we aim to practice continuous delivery when we’re able to, this isn’t always the case for more complicated situations. For data that falls under legal regulations, you likely will need to use either a manual process or a schedule-based one by running a cron job on the server. For non-private data, you can use the merge/push-based process by using a GitHub workflow.

Regardless of the underlying trigger, the actual release process is the same: a release is created based on specific text within the commit messages. Using commit messages to determine a release is called semantic release, which uses Semantic Versioning and Conventional Commits as its foundation.

11.1.1 Commits

In order to determine whether changes should result in a release, commits must follow Conventional Commits to structure the commit messages. The structure of a commit message looks like this:

<type>(optional scope): <optional-emoji> <description>

[optional body]

[optional footer(s)]

The two main components of this structure for determining releases are the “type” and the “footer”. The “type” is the first part of the commit message and is used to determine what type of change has been made. The “footer” is the last part of the commit message and is used to include a BREAKING CHANGE note if the change is a breaking change, i.e., a change that disrupt the compatibility of existing code with newer versions. A breaking change can also be indicated by appending a “!” to the commit “type”. These two components determine which version to set for the release.

Semantic versions are made up of three numbers: MAJOR.MINOR.PATCH, e.g., 0.1.2. The MAJOR version is incremented when there are breaking changes, the MINOR version is incremented when new features are added, and the PATCH version is incremented when fixes are made. For semantic releases, the commit “type” feat increases the MINOR version, while the commit “type” fix, refactor, or perf increases the PATCH. If there is a breaking change, with either BREAKING CHANGE in the footer or <type>! in the commit message, then the MAJOR version is increased.

But how do you know which commit type to use? Unlike software development, developing data packages is quite different and it can be a bit more difficult to determine what a “feature”, “fix”, or “breaking change” is. To help determine the commit “type”, we use aspects of Data Package’s semantic versioning guide.

Breaking changes with the <type>! or BREAKING CHANGE in the footer format must only happen after the first stable release of the data package. The first stable release is defined as when the data package has all expected or planned resources, metadata has been completed, and all “observational units” (e.g. participant or animal) have been measured. Essentially, when the planned study has been completed. Before that point, only MINOR and PATCH changes are allowed. This means that the version will remain at 0.MINOR.PATCH until the stable release. Once a stable release has been made, a breaking change would occur if you:

  • Change the data package, resource, or column name or identifier.
  • Remove a resource or column from the data package.
  • Move a column into another resource.
  • Change a column type (e.g. from integer to string).
  • Change a column’s constraints to be more restrictive (e.g. reduce the distance between the minimum and maximum values).
  • Remove a participant’s data (e.g. after they request their data be deleted).
  • Substantially change the meaning of the text in the metadata (e.g. a column’s description or a resource’s title).

A good guideline to use for MINOR (feat) commits would be if something new has been added or expanded on. Minor changes with the feat format would be if you:

  • Add a new resource.
  • Add data, either new rows or columns to an existing resource.
  • Change a column’s constraints to be less restrictive (e.g. increase the distance between the minimum and maximum values).
  • Add new text to the metadata, for example, when no metadata existed before, but not correcting existing metadata (see PATCH below).

A good guideline to use for PATCH (fix, refactor, or perf) commits would be if something has been corrected or refined. Before the stable release, many of the breaking change items above would be considered a patch change, as they generally don’t add any new content. Patch changes with the fix, refactor, or perf commit type would be if you:

  • Correct errors in existing data, like a typo or data entry error. Depending on the severity of the error, this could also be a breaking change.
  • Change the text of the metadata without changing the meaning, for example fixing typos, grammatical errors, or clarifying the text without changing its meaning.
  • Changes to how the data is processed so that it results in better compression or other performance improvements.

Any other commit types can be used, but they won’t result in a releasable change. For example, before any metadata or resources have been created as the pipeline is being developed, you could use the chore commit type to indicate that you are working on the pipeline, but no data or metadata has been created yet.

11.2 Steps

Now that we’ve covered the triggers, let’s go over the actual steps involved in the release process, whether it is manual, schedule-based, or merge/push-based. Cocogitto manages all these steps via the cog.toml file.

The release process runs the following steps to check and potentially update the release number:

  1. Check the commit history since the last release for any releasable changes. If no releasable changes are found, then no release is created. Otherwise, the process continues.

  2. Update the version based on the commit message and update the version in the pyproject.toml file using uv version. If the metadata format is datapackage.json, the version field uses the version in pyproject.toml and will be updated automatically when the datapackage.json file is (re)generated.

  3. Run the build process from start to end, excluding the step to pull from the sources and saving to raw/. Raw data is only saved intentionally as a separate commit/pull request. This is described in the build process in

    1. The main artifacts of the build process are the <package-name>_<version>.tar file for sensitive and nonsensitive data and the <package-name>_<version>.zip for an artifact that can be publicly uploaded when the data is sensitive.

  4. Generate the changelog based on the commit messages since the last release. git-cliff is used to generate the changelog.

  5. Commit the changes that were made in the CHANGELOG.md file and the metadata file (e.g. datapackage.json), then create a tag for the new version on that commit. No data in raw/ is committed. Push the commit and the tag to GitHub.

  6. Create a new GitHub release on GitHub from the new tag and changelog. Attach the build artifacts to the release. For nonsensitive data and that is smaller than 2 Gb (GitHub’s limit), the .tar file is attached to the release. For sensitive data or for data that is larger than 2 Gb, the .zip file is used instead. Either way, the file is renamed to simply <package-name>.zip (or <package-name>.tar), as the tag itself contains the version number.

  7. For nonsensitive data and that is smaller than 50 Gb (the limit for Zenodo), upload the <package-name>.tar file to Zenodo. For sensitive data or data that is larger than 50 Gb, upload the <package-name>.zip file instead.

Important

For data that is larger than GitHub’s 2 Gb limit or Zenodo’s 50 Gb limit, the best place to store the released .tar file is in your server in the releases/ folder. You’ll likely need to delete older releases if space becomes an issue on the server. Deleting older releases is fine, as the release process is designed to be reproducible, so you can always recreate the release by switching to the tagged commit and running the build process again.

11.3 Practical considerations

As you develop a data package, there are a few things to keep in mind in order to make the release process easier.

  • We consider the first, non-stable (< 1.0.0) release to happen once there is code that takes the first resource and its metadata from raw format into its final resource state. The code must also be integrated into the build.py file, so that the release can be reproducible.

  • Whenever you make a change, either directly to main or through a pull request, you always need to make sure commits and pull requests are atomic. This means that each commit or pull request contains only one conceptual change. That’s because the commit message (and consequently the changed files in the commit) determine what type of release will be created. The commit message will also be added to the changelog, so be aware of the message you use.