Use SPM to store dependency checkouts in a repository and do it better than CocoaPods
Many of you have experienced the annoying situation — you open a project or switch a branch, and see the sad picture of how SPM resolves packages.
One of the advantages of CocoaPods, compared to SPM, is that dependency checkouts are stored with the project directly in the repository. This allows you to painlessly launch project from any commit and not waste time on CI to download dependencies and resolve them.
In this article, I will show you how to use SPM to store dependency checkouts in a repository and do it better than CocoaPods.
Before we start, let’s define a list of requirements for a future solution:
- We continue to live in the paradigm of Swift packages
- External package dependencies become local
- Requires a mechanism to locally clone external dependencies
- Store only files from dependency repositories locally that are required by the project
With the inputs in hand, we can start implementing.
Let’s start with cloning dependencies. To do this, we will create a separate local package and add all the required dependencies to Package.swift:
// swift-tools-version: 5.7
// The swift-tools-version declares the minimum version of Swift required to build this package.
import PackageDescription
let package = Package(
name: "VendorPackages",
platforms: [
.iOS(.v15),
.macOS(.v12),
],
products: [
.library(
name: "VendorPackages",
targets: ["VendorPackages"]
),
],
dependencies: [
.package(
url: "https://github.com/devicekit/DeviceKit.git",
exact: "5.0.0"
),
.package(
url: "https://github.com/kean/Nuke.git",
exact: "12.1.2"
),
.package(
url: "https://github.com/groue/GRDB.swift.git",
exact: "6.15.1"
),
.package(
url: "https://github.com/SnapKit/SnapKit",
exact: "5.6.0"
),
.package(
url: "https://github.com/airbnb/lottie-ios",
exact: "4.2.0"
),
],
targets: [
.target(
name: "VendorPackages",
dependencies: [
"DeviceKit",
.product(name: "Nuke", package: "Nuke"),
.product(name: "NukeUI", package: "Nuke"),
.product(name: "NukeExtensions", package: "Nuke"),
"SnapKit",
.product(name: "GRDB", package: "GRDB.swift"),
.product(name: "Lottie", package: "lottie-ios"),
]
),
]
)
This package contains no code. Its only value is the dependency manifest.
After opening the package in Xcode or running the swift package resolve command, notice that a hidden .build directory has appeared in the package directory with a checkouts subdirectory that contains local dependency clones:
Let me draw your attention to the size of the .build directory. It is equal to 993 MB, of which 328 MB is occupied by checkouts and 665 MB by repositories. I have deliberately selected popular packages (GRDB, Lottie) with a long history to demonstrate the scale of the problem.
Knowing where the local dependency clones are located, our next step is to determine the minimum files from these packages that our project needs to build. Such files are obviously the target sources of those products that are used in our original local package, where we specified the required external dependencies.
We are absolutely not interested in test target files, or various demo resources. In other words, everything that just takes up disk space and is not used when compiling the project. If necessary, these “extra” files can always be used separately.
Determining this minimum of files may at first glance seem like a rather non-trivial task, but in practice, the SPM CLI provides us with everything we need:
swift package describe --type=json
Let’s use this command in the directory of, for example, the Nuke package:
{
"dependencies" : [
],
"manifest_display_name" : "Nuke",
"name" : "Nuke",
"path" : ".build/checkouts/Nuke",
"platforms" : [...],
"products" : [...],
"targets" : [
{
"c99name" : "NukeVideo",
"module_type" : "SwiftTarget",
"name" : "NukeVideo",
"path" : "Sources/NukeVideo",
"product_memberships" : [
"NukeVideo"
],
"sources" : [
"AVDataAsset.swift",
"ImageDecoders+Video.swift",
"VideoPlayerView.swift"
],
"target_dependencies" : [
"Nuke"
],
"type" : "library"
},
{
"c99name" : "NukeUI",
"module_type" : "SwiftTarget",
"name" : "NukeUI",
"path" : "Sources/NukeUI",
"product_memberships" : [
"NukeUI"
],
"sources" : [
"FetchImage.swift",
"Internal.swift",
"LazyImage.swift",
"LazyImageState.swift",
"LazyImageView.swift"
],
"target_dependencies" : [
"Nuke"
],
"type" : "library"
},
{
"c99name" : "NukeExtensions",
"module_type" : "SwiftTarget",
"name" : "NukeExtensions",
"path" : "Sources/NukeExtensions",
"product_memberships" : [
"NukeExtensions"
],
"sources" : [
"ImageLoadingOptions.swift",
"ImageViewExtensions.swift"
],
"target_dependencies" : [
"Nuke"
],
"type" : "library"
},
{
"c99name" : "Nuke",
"module_type" : "SwiftTarget",
"name" : "Nuke",
"path" : "Sources/Nuke",
"product_memberships" : [
"Nuke",
"NukeUI",
"NukeVideo",
"NukeExtensions"
],
"sources" : [/* a lot of sources here */],
"type" : "library"
}
],
"tools_version" : "5.6"
}
JSON output contains an array of targets, which literally have all the necessary source files, as well as products that include these targets (if the target contains resources, they will be located by the “resources” key).
Boom! 100% what we need.
Since the most complex and non-trivial part turned out to be trivial and can be solved with a single command, then all that remains at this stage is to write a small script that will copy the necessary files for the products specified in our local project, preserving the directory structure.
I did it with a simple shell script and jq utility. You are free to use the tools familiar to you, relying on my implementation as a reference:
rm -rf Sources && mkdir -p Sources
cd _Proxy # directory for local package with remote dependencies
swift package clean
swift package update
required_products=$(swift package describe --type json | jq -c '(.targets[] | select(.name=="VendorPackages")) | .product_dependencies[]')
for repo in $(ls .build/checkouts); do
echo $repo
mkdir -p ../Sources/$repo
cp -r .build/checkouts/$repo/Package.swift ../Sources/$repo/_Package.swift
package_json=$(swift package --package-path .build/checkouts/$repo describe --type json | jq -c)
targets=$(jq -c '(.targets[] | select(.product_memberships != null))' <<< $package_json)
echo "$package_json" | jq -c '(.targets[] | select(.product_memberships != null))' | while read -r target; do
required_target=false
target_products=$(jq -c '.product_memberships[]' <<< "$target")
for target_product in $target_products; do
for required_product in $required_products; do
if [ $required_product == $target_product ]; then
required_target=true
fi
done
done
if ! $required_target; then
continue
fi
name=$(jq -r '.name' <<< $target)
path=$(jq -r '.path' <<< $target)
type=$(jq -r '.type' <<< $target)
if [ $type == "system-target" ]; then
mkdir -p ../Sources/$repo/$path
cp -r .build/checkouts/$repo/$path/. ../Sources/$repo/$path
fi
if [ $type == "library" ]; then
echo "$target" | jq --raw-output '.sources[]' | while read -r source; do
mkdir -p ../Sources/$repo/$path/"$(dirname "$source")"
cp .build/checkouts/$repo/$path/"$source" ../Sources/$repo/$path/"$source"
done
fi
done
done
Using this script, we copied all the files needed to build the project from all external dependencies.
The last question that needs to be answered is — how to actually use the copied files in our project.
There is no single answer. For example, you can expand the script and copy package manifests along with other files, somehow removing unused targets from there: using regular expressions, AST modifications, etc.
I chose not to automate this step and added another local package, where I registered all the dependencies, specifying the local paths to the copied files:
// swift-tools-version: 5.7
// The swift-tools-version declares the minimum version of Swift required to build this package.
import PackageDescription
let package = Package(
name: "VendorPackages",
platforms: [
.iOS(.v15),
.macOS(.v12),
],
products: [
.library(name: "DeviceKit", targets: ["DeviceKit"]),
.library(name: "GRDB", targets: ["GRDB"]),
.library(name: "Lottie", targets: ["Lottie"]),
.library(name: "Nuke", targets: ["Nuke"]),
.library(name: "NukeUI", targets: ["NukeUI"]),
.library(name: "NukeExtensions", targets: ["NukeExtensions"]),
.library(name: "SnapKit", targets: ["SnapKit"]),
],
targets: [
.target(
name: "DeviceKit",
path: "Sources/DeviceKit/Source"
),
.target(
name: "Nuke",
path: "Sources/Nuke/Sources/Nuke"
),
.target(
name: "NukeUI",
dependencies: ["Nuke"],
path: "Sources/Nuke/Sources/NukeUI"
),
.target(
name: "NukeExtensions",
dependencies: ["Nuke"],
path: "Sources/Nuke/Sources/NukeExtensions"
),
.target(
name: "GRDB",
dependencies: ["CSQLite"],
path: "Sources/GRDB.swift/GRDB"
),
.systemLibrary(
name: "CSQLite",
path: "Sources/GRDB.swift/Sources/CSQLite"
),
.target(
name: "Lottie",
path: "Sources/lottie-ios/Sources"
),
.target(
name: "SnapKit",
path: "Sources/SnapKit/Sources"
),
],
swiftLanguageVersions: [.v5]
)
The advantage of this approach is that you control the description of the targets, their settings, and so on.
Thus, we have automated the granular copying of the required dependency files but left the local package manifest that will be used in the project under manual control.
Unbelievable, but true: the size of the copied files is 4.7 MB — which is 211 times smaller than our .build directory.
You can find the source code for the article in this repository.
Happy coding!
How to Use Swift Package Manager to Save Gigabytes of Network Traffic and Disk Space was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.