Improve build times by extracting 3rd party tooling to processing script.



A lot has been written about improving Swift compile times, but the compiler and linker are just part of the equation that slows down our development cycle.

Most projects leverage great 3rd party tools like SwiftLint, Sourcery, SwiftGen, SwiftFormat and many more. Leveraging those tools is the right thing to do but we have to be aware that all of those tools come with some time delay for our build -> run development cycle.

We often set those tools to run as build-phases which means they run each time you attempt a build but none of those tools need to be run each time we build.

Even tools that generate code we need for our projects like Sourcery or SwiftGen don’t need to be re-run unless we made changes in very specific parts of the application.

As an example New York Times main application leverages a lot of 3rd party and internal tooling, the total time all the tools take is 6s on my (powerful) machine.

Only 6 seconds or as much as 6 seconds? Let’s put this into context:

  • I build a project between 200-400 times on an average workday.
  • Let’s assume 90% of the time we don’t need to run those tools with each build.
  • We have 30 iOS developers working on the main app

Lower limit: 200 * 6s * 30 * 90* => 9 hours wasted per day

We are wasting 45 hours per week, if we can improve that it’s almost like hiring a new full-time developer, except it’s free.

Let’s look at how we can improve this with a process change a dash of bash shell programming.

Establishing a new process

If we are not going to run those tools each build we need to guarantee a few things:

  • We need a way to run all our tooling manually and it needs to be consistent since we have a lot of different tools
  • Our repository should never be in a bad state, so no code can exist at any point that is not processed by our tooling
  • Since we should never trust humans when a machine can be used, we are going to make sure CI enforces the repository state

If we try commiting and forget to run the tool we’ll get an error like (here using GitUp IDE):

Implementation

We are going to create a shell script that can be run manually by calling ./Scripts/process.sh. It will also be a part of our pre-commit hook

First, let’s set the script up with 2 modes:

  • fail-on-errors -> any error will cause the script to exit with failure code, used by CI / pre-commit
  • local -> manual developer runs, we don’t fail script and we can use this mode to add some console coloring & extra debug information if need be
#!/bin/zsh

if [[ -n "$CI" ]] || [[ $1 == "--fail-on-errors" ]] ; then
 FAIL_ON_ERRORS=true
 echo "Running in --fail-on-errors mode"
 ERROR_START=""
 COLOR_END=""
 INFO_START=""
else
 echo "Running in local mode"
 ERROR_START="\e[31m"
 COLOR_END="\e[0m"
 INFO_START="\e[34m"
fi


Next, let’s configure some required variables for our project and the most common tooling:

final_status=0

# :- is like swift's ?: so default fallback value
PODS_ROOT=${PODS_ROOT:-"Pods"}
PROJECT_DIR=${PROJECT_DIR:-$(pwd)}
# Needed for SwiftGen
export PRODUCT_MODULE_NAME=${PRODUCT_MODULE_NAME:-"OurProjectModuleName"}

if [[ `xcode-select -p` =~ CommandLineTools ]] ; then 
 echo "${ERROR_START}Your toolchain won't run Swiftlint or Sourcery. Use\n sudo xcode-select -s /path/to/Xcode.app\nto fix this.${COLOR_END}"
fi

Checking if an external tool has done anything to our project

Now we are ready to start running our tools but the issue we’ll immediately run into is the fact that each tool might be doing its job in many different ways:

  • Files might be added/changed e.g. SwiftGen, Sourcery
  • Messages might be printed on stdout e.g. SwiftLint
  • Only the status code will be returned

How can we account for all of that?

File changes

We can leverage git diff to verify if anything was changed in our repo:

# execute like this `process name command`
function process()
 local initial_git_diff=`git diff --no-color`
 eval "$2"
 if [ "$FAIL_ON_ERRORS" = "true" ] && [[ "$initial_git_diff" != `git diff --no-color` ]]
 then
 echo "${ERROR_START}$1 generates git changes, run './Scripts/process.sh' and review the changes${COLOR_END}"
 final_status=1
 fi
}

process "Sourcery" "${PODS_ROOT}/Sourcery/bin/sourcery --prune --quiet"

Standard output

Our processing function simply needs to see if output was generated:

local output=$(eval "$2")
if [[ ! -z "$output" ]]
 then ...

process_output "SwiftLint" "${PODS_ROOT}/SwiftLint/swiftlint lint --quiet"

Status code

Simply checking the status code after executing the command should suffice:

eval "$2"
local return_value=$?

process_return_code "MyCustomRubyScript" "ruby Scripts/myScript.rb MyProject.xcodeproj"

Final tweaks

Colors and timing

Let’s improve information printed by the script by adding timing and coloring to our process functions:

function process() {
 echo "\n${INFO_START}# Running $1 #${COLOR_END}"
 local initial_git_diff=`git diff --no-color`
 local start=`date +%s`
 
 eval "$2"

 if [ "$FAIL_ON_ERRORS" = "true" ] && [[ "$initial_git_diff" != `git diff --no-color` ]]
 then
 echo "${ERROR_START}$1 generates git changes, run './Scripts/process.sh' and review the changes${COLOR_END}"
 final_status=1
 fi

 local end=`date +%s`
 echo Execution time was `expr $end - $start` seconds.
}

Final status check

if [[ $final_status -gt 0 ]]
then
 echo "\n${ERROR_START}Changes required. Run './Scripts/process.sh' and review the changes${COLOR_END}"
fi

exit $final_status

Connecting to pre-commit and CI

In our pre-commit and CI scripts we simply run it in failure mode:

./Scripts/process.sh --fail-on-errors
failed=$?
exit $failed

Summary

Final script you can use for your workflow

We have been using this approach for a few years now and it has saved us thousands of work-hours

During normal development, we don’t run those processing tools, when we change code that requires them we simply run it from the command line (or desktop shortcut if you are into that).

We can sleep peacefully because when we try to commit code our pre-commit hook runs it for us and lets us know if there were any changes that we need to review before proceeding. Even if someone messes up their pre-commit hook setup there is always CI ensuring they fix it.




If you hate writing repetitive code SourceryPro is here.

Want to optimize your development workflow? I'd love to help your team.