flowchart TB
Source["source<br>(e.g., API<br>or database)"] -->|pull| Raw["raw/"]
Raw -->|process| Staging["staging/"]
Staging -->|join| Resources["resources/"]
Resources
Metadata["metadata<br>(Optionally<br>regenerated)"]
Metadata ---|check| Staging
Metadata & Resources -->|bundle| Build["releases/<br>.tar and .zip"]
10 Build process
Like software packages (e.g. in Rust, Python, R), the Git repository contains the source of the package rather than the package itself. It includes the raw input data and the code needed to process it into the final format. But before the data package can be distributed, it must first be “built” (just like code is “compiled”) by running the code to transform the raw data inputs to the final packaged format.
At a high level, the directories and files involved in the build process are (non-relevant files and directories are removed):
<package-name>/
├── .github/
│ └── workflows/
│ └── release.yml # Optional, depends on legal requirements
├── raw/
├── staging/
├── resources/
├── releases/
├── src/
│ └── <package_name>/
│ ├── __init__.py
│ ├── build_raw.py
│ └── build_staging.py
├── <metadata-format>.<extension>
├── CHANGELOG.md
├── README.md
├── LICENSE.md
└── <build-files>
Note
Why do we “build” a data package?
- To keep a separation between the source code and raw input data contained within the Git repository and the final data package that is released.
- To ensure a reproducible build from input to output.
- To keep a history of the changes made to the final data and to version the data package for easier tracking of changes and updates.
- To treat the data package as a formal “product” and apply rigorous and robust software engineering and data engineering practices to its development and release.
10.1 Steps
The initial steps, from the source of the data to staging, is managed by a pipeline management tool, which should have its pipeline defined in src/<package_name>/build_*.py (though some pipeline tools require a different filename). We recommend using pytask, which we’ve found works well for developing data packages (for more details, see our pytask for Python workflow management decision post). All steps must be defined within this tool, from pulling the source data to raw/, to processing the raw data into staging/, and to processing or extracting any metadata obtained from the source or staged data. Because data saved into raw/ is committed in separate pull requests (described more below and in Chapter 11), there should be two build steps: downloading data from source locations to raw/ and taking the data in raw/ and processing it into staging/. We suggest calling these files build_raw.py and build_staging.py, so that they can be run independently of each other.
The steps after staging/ are managed mainly by Sprout, which is our tool to handle the processing from staging/ to resources/ and for (re-)generating the metadata file.
All the steps for building the data package into .tar and .zip files are handled through the justfile file, which would be (one of) the <build-files> in the above directory structure. The justfile, if you use our template data package, contains recipes for each step of the build process.
If the data package is based on open data or on data not covered by legal restrictions (e.g. GDPR), the build and release process can be automated through a GitHub Actions workflow defined in .github/workflows/release.yml. Otherwise, the process needs to be done manually or through a scheduled workflow on a secure server. We cover this more in Chapter 11.
Note
We plan on building a tool to automate and simplify the build steps, but for now, you can use the structure and tools that we’ve set up in our Template Data Package. See Chapter 13 for details about that.
Important
It’s important to note here that only the data in raw/ and the metadata file are saved into Git LFS during the release process. No other data artifacts or files are saved in the Git history. This is described in more detail in Chapter 11.
10.1.1 Pull from source to raw
To make sure we’re getting the most up-to-date data, the first build step is to pull from the various data sources into raw/. See Chapter 5 and Chapter 4 for details on how the raw data is organised and how the source code pulls the data via the pipeline tool. The data is not processed in any way when it is pulled from the source into raw/.
10.1.2 Raw to staging
After pulling the source data into the raw/ directory, the data (usually) needs to be processed in some way. In particular, the data likely needs to be reorganised so that it ends up in the correct resource location. The data in raw/ should be processed into staging/ in a one-to-one mapping, so that every <timestamp>.<extension> file in raw/ is processed into a <timestamp>.parquet file in staging/. This enables the potential for parallel processing and to maintain a sequence between the two directories. Any data in staging/ must also match the contents of the metadata format. This is all done to ensure that the data gets into the final resource correctly and without issues. See Chapter 6, Chapter 8, and Chapter 4 for more details staging/.
10.1.3 Staging to resources
At this stage, Sprout can take over and process the data from staging/ into resources/. All the timestamped files in the individual staging/ resource directories are processed and joined into a single resource file in resources/. During processing, the data is checked against the metadata. See Chapter 7, Chapter 8, and Sprout’s documentation for more details on this step.
10.1.4 Metadata (re-)generation (optional)
For data packages using datapackage.json as the metadata format, Sprout will also (re-)generate the file from the Python code. See Chapter 8 and Sprout’s documentation for a description of this section.
10.1.5 Build artifacts
At the end, the data package is built into one <package-name>_<version>.tar file that contains the metadata file (e.g. datapackage.json), LICENSE.md, README.md, CHANGELOG.md, and the resource files. If the data contains human (especially health) data and if it is required to be on secure servers, an additional <package-name>_<version>.zip file is also built with the same files but does not include the data. This .zip file can be uploaded to public archives to generate, for example, a DOI on Zenodo, while the .tar file can remain on the server. The .tar and .zip files are saved into a releases/ directory ignored by Git.
10.2 Development practices
How does this build process impact developing the data package and the development practices? Because of this formal build process, there are a few things to be aware of.
First, you should only save raw data in the Git history within the Git LFS (after pulling it from the sources) during an intentional update. A pull request (or commit) should only contain changes to raw/ in that commit, meaning the commit should be atomic as a data update. You should do it this way because you need to know what has changed in the raw data to be able to correctly use the right Conventional Commit type (see Chapter 11 for more details). For example, if the raw data update has completely new data, the type would be feat, but if it was correcting a data entry error, it would be fix.
Related to the above, outside of these intentional and atomic commits/pull requests, you shouldn’t commit the raw data to the Git history for any other change, such as when changing the code. During these types of changes, treat any data pulled from sources into raw/ or processed into staging/ or resources/ as temporary.
Related to the above, if the metadata format is datapackage.json, pull requests should not contain any changes to it. These files are generated during the build process and should not be modified or added directly. For datapackage.json based metadata, the metadata files within src/ that contain the metadata managed by Sprout should be modified instead. For other metadata formats, the metadata file can be modified directly and can be saved to the Git history.
Commit messages should still be written in the Conventional Commits format, though the specific commit types used are a bit different considering no data, or metadata files if it is the datapackage.json format, are being modified or saved directly. See Chapter 11 for more details on how Conventional Commits are used in the release process and what commit messages to use.
All files in staging/, resources/, and releases/ should be ignored by Git in the .gitignore file, aside from a .gitkeep or README.md file to keep the directory structure.