Let’s Parse Xcode Logs?
In this article, we will try to understand how and where Xcode stores its logs, what SLF0 is, how to read it, maybe even understand it, and see how it’s better and more interesting without an IDE.
No extra preamble.
To see the build logs of a project or tests in normal, everyday development, we go and look at the last tab of the Xcode toolbar.
The organization of the log as a whole is readable; it is even possible to export all this to the txt format.
But, all this won’t help us automate metrics collection. For example, it would be hard if we wanted to accumulate the dynamics of module assembly time while running a job on CI.
There is no easy way to export this entire log to interpret specific events.
So, Xcode collects and stores all logs in DerivedData/{ProjectName}/Logs.
What is DerivedData?
By default, it is located in the following path:
open ~/Library/Developer/Xcode/DerivedData
This path can be changed in the Xcode settings:
Actually, DerivedData stores all intermediate information (including logs), as well as artifacts required to optimize the subsequent build.
Inside the Logs folder, you can find many logs of various stages that we encounter during development:
Here we can find the following kinds of files:
- .xcactivitylog — the actual log, which will be of interest to us later
- .xcresult — bundle that stores summary information. You can open it in Xcode and see the logs as if we just opened the last tab of the toolbar
- LogStoreManifest.plist— contains summary information in a readable form about all xcactivitylog files. They can be used, for example, to monitor the readiness of the next log with logs.
Parsing xcactivitylog
So, the log file is a gzip-compressed log, and the name is a unique UUID.
After unpacking, we get a log written in the SLF0 serialization format.
Here’s an example of an unpacked log in SLF0 format:
SLF010#21%IDEActivityLogSection1@0#39"Xcode.IDEActivityLogDomainType.BuildLog25"Build XCActivityLogParser212"eyJ0eXBlIjp7ImJsdWVwcmludFByb3ZpZGVyIjp7fX0sImJsdWVwcmludFByb3ZpZGVyX3Byb3ZpZGVyRmlsZVBhdGhTdHJpbmciOiJcL1VzZXJzXC92b3JvYnlvdlwvRG9jdW1lbnRzXC9YQ0FjdGl2aXR5TG9nUGFyc2VyXC9YQ0FjdGl2aXR5TG9nUGFyc2VyLnhjb2RlcHJvaiJ9cbbac35a7833c541^0000007fc3632d42^1(1@1#27"com.apple.dt.IDE.LogSection16"Prepare packages212"eyJ0eXBlIjp7ImJsdWVwcmludFByb3ZpZGVyIjp7fX0sImJsdWVwcmludFByb3ZpZGVyX3Byb3ZpZGVyRmlsZVBhdGhTdHJpbmciOiJcL1VzZXJzXC92b3JvYnlvdlwvRG9jdW1lbnRzXC9YQ0FjdGl2aXR5TG9nUGFyc2VyXC9YQ0FjdGl2aXR5TG9nUGFyc2VyLnhjb2RlcHJvaiJ9f73bc45a7833c541^4303c95a7833c541^---0#0#0#46"Compile plug-ins and run any prebuild commands--36"52BE500F-D551-461D-975D-BF4B4AA236BF--0"0(0#0#0#---36"2976A337-D8BA-4626-B5F2-41F0F7CB232E--
No official documentation on the web describes this type of serialization. Everything we managed to find is the fruit of the work of enthusiasts, for which many thanks go to them.
Each log starts with the obligatory SLF0 header.
The following is a sequence of tokens in the following format:
<side_value><delimiter><side_value>
<side_value> is an optional value and depends on the specific separator.
The current version of SLF0 supports seven delimiters, each of which is a description of a value of a specific type:
# integer
Describes an integer type, for example:
10#
Left value: UInt64, the value of the token itself.
Right value: missing.
“string”
Describes the string type, for example:
21"IDEActivityLogSection
Left value: UInt64 describes the number of characters that come to the right of the delimiter; it is also the value of the string token. In this case, that means the string value you’re looking for is the string on the right, which is 21 characters long. The character count is in utf-16 and does not match the default count for instances of the string type, which returns the number of graphemes.
Right value: A string consisting of the number of characters to the left of the delimiter.
^ double
Describes the double type. Here’s an example:
4fcbedd82b32c541^
Left value: is a big-endian floating point number encoded in hexadecimal. Used in the log to represent the timeInterval type.
Here’s an example of casting to a regular double from the Swift language:
guard let integerValue = UInt64("4fcbedd82b32c541", radix: 16) else { exit(0) }
let double = Double(bitPattern: integerValue.byteSwapped)
print(double) // 711219121.857767
Right value: missing.
– null
Describes a null value, i.e., we simply do not have the value of any field. Here’s an example:
-
Left value: missing.
Right value: missing.
( array
Describes an array and indicates the number of elements it contains. Here’s an example:
2(
Left value: the number of elements in the array. The array elements are values of the class_instance type, which will be discussed below.
Right value: missing.
% class_name
Describes class_name. In SLF0, the order of all tokens in the log is the decisive success factor; it is important to keep their order when parsing. The resulting indexing class_name is the one we are looking for when parsing class_instance.
38%IDEActivityLogCommandInvocationSection
Left value: everything here is by analogy with the string type, which is described above.
Right value: class name.
@ class_instance
Describes class_instance, i.e., points to the name of the class, based on which it should become clear how to interpret all the tokens that will follow this class_instance.
1@
Left value: UInt64, the actual value of the token, indicating the position of the corresponding class_name in the log (required to understand what type the tokens that follow are).
Right value: missing.
For the convenience of understanding this serialization language, I outlined a description of the syntax in the form of the Backus-Naur form (it may be useful when developing the corresponding parser):
<header> := SLF0
<integer_delimiter> ::= #
<double_delimiter> ::= ^
<null_delimiter> ::= -
<string_delimiter> ::= "
<array_delimiter> ::= (
<class_name_delimiter> ::= %
<class_instance_delimiter> ::= @
<delimiter> ::= <integer_delimiter>|<double_delimiter>|<null_delimiter>|<string_delimiter>|<array_delimiter>|<class_name_delimiter>|<class_instance_delimiter>
<side_value> ::= <character>|<digit>|<side_value><character>| <side_value><digit>
<value> ::= [<side_value>]<delimiter>[<side_value>]
<log> ::= <value>|<log><value>
<all_log> ::= <header><log>
I didn’t describe what <digit> and <character> are; I think it’s clear.
So, knowing all the above syntactic and semantic rules, let’s try to read some specific logs (from the example above). After reading, we get the following list of tokens, one by one (I use the written tokenizer):
Here’s an example of received tokens xcactivitylog:
integer(10)
class_name("IDEActivityLogSection")
class_instance(1)
integer(0)
string("Xcode.IDEActivityLogDomainType.BuildLog")
string("Build XCActivityLogParser")
string("eyJ0eXBlIjp7ImJsdWVwcmludFByb3ZpZGVyIjp7fX0sImJsdWVwcmludFByb3ZpZGVyX3Byb3ZpZGVyRmlsZVBhdGhTdHJpbmciOiJcL1VzZXJzXC92b3JvYnlvdlwvRG9jdW1lbnRzXC9YQ0FjdGl2aXR5TG9nUGFyc2VyXC9YQ0FjdGl2aXR5TG9nUGFyc2VyLnhjb2RlcHJvaiJ9")
double(711389365.529138)
double(63113904000.0)
array(1)
class_instance(1)
integer(1)
string("com.apple.dt.IDE.LogSection")
string("Prepare packages")
string("eyJ0eXBlIjp7ImJsdWVwcmludFByb3ZpZGVyIjp7fX0sImJsdWVwcmludFByb3ZpZGVyX3Byb3ZpZGVyRmlsZVBhdGhTdHJpbmciOiJcL1VzZXJzXC92b3JvYnlvdlwvRG9jdW1lbnRzXC9YQ0FjdGl2aXR5TG9nUGFyc2VyXC9YQ0FjdGl2aXR5TG9nUGFyc2VyLnhjb2RlcHJvaiJ9")
double(711389365.53308)
double(711389365.570412)
null
null
null
integer(0)
integer(0)
integer(0)
string("Compile plug-ins and run any prebuild commands")
null
null
string("52BE500F-D551-461D-975D-BF4B4AA236BF")
null
null
string("")
array(0)
integer(0)
integer(0)
integer(0)
null
null
null
string("2976A337-D8BA-4626-B5F2-41F0F7CB232E")
null
null
At first glance, everything is very scary and no less clear than before. However, let’s try to figure it out.
To begin with, please pay attention to the class names. In fact, there are a limited number of them at the time of this writing:
- IDEActivityLogSection
- IDEActivityLogCommandInvocationSection
- IDEActivityLogUnitTestSection
- IDEActivityLogMajorGroupSection
- IDEActivityLogMessage
- DVTDocumentLocation
- DVTTextDocumentLocation
When trying to find information about the above types, it becomes clear that they are all described in the private framework IDEFoundation.framework, which is part of Xcode and is located at:
open /Applications/Xcode.app/Contents/Frameworks/IDEFoundation.framework/Versions/A/
Further, through reverse engineering, you can find all these classes and reproduce their set of fields. After that, you can compare them with the values of the tokens obtained above (all manipulations are for research purposes).
Out of nothing but pure interest, I dug around using ktool and was able to get a description of all classes without much difficulty:
ktool dump --headers /Applications/Xcode.app/Contents/Frameworks/IDEFoundation.framework/Versions/A/IDEFoundation
Below, you’ll see an example of IDEActivityLogSection. For convenience, let’s only look at its initializer, which reflects all the fields we need:
-(id)initWithSectionType:(NSInteger)arg0
domainType:(id)arg1
title:(id)arg2
subtitle:(id)arg3
location:(id)arg4
signature:(id)arg5
timeStartedRecording:(CGFloat)arg6
timeStoppedRecording:(CGFloat)arg7
subsections:(id)arg8
text:(id)arg9
messages:(id)arg10
wasCancelled:(char)arg11
wasFetchedFromCache:(char)arg12
commandDetailDescription:(id)arg13
resultCode:(NSInteger)arg14
uniqueIdentifier:(id)arg15
localizedResultString:(id)arg16
xcbuildSignature:(id)arg17
The only problem is that the order of the tokens don’t match the order obtained by researching the framework.
However, the people from the XCLogParser tool have already found and compared everything. Many thanks to them. You can see how all the received types look with the correct field order here.
For example, here’s all of the fields in IDEActivityLogSection; they’re in the correct order:
public let sectionType: Int8
public let domainType: String
public let title: String
public let signature: String
public let timeStartedRecording: Double
public var timeStoppedRecording: Double
public var subSections: [IDEActivityLogSection]
public let text: String
public let messages: [IDEActivityLogMessage]
public let wasCancelled: Bool
public let isQuiet: Bool
public var wasFetchedFromCache: Bool
public let subtitle: String
public let location: DVTDocumentLocation
public let commandDetailDesc: String
public let uniqueIdentifier: String
public let localizedResultString: String
public let xcbuildSignature: String
Knowing all this, let’s try to decrypt the received tokens:
The first token is always the SLF0 serialization format version. In our case, it’s the tenth one:
integer(10) // Serialization format version
Next comes the first declaration of the type name that tokens of type class_instance will refer to:
class_name("IDEActivityLogSection") // Class name (needed to understand what type the fields of class_instance belong to)
The description of the first object begins with the values of its parameters. class_instance(1) indicates that the following tokens should be interpreted as class_name, which is specified under the corresponding relative index 1. In our case, this is what IDEActivityLogSection looks like:
class_instance(1) // Specifies that an object of type IDEActivityLogSection
integer(0) // sectionType
string("Xcode.IDEActivityLogDomainType.BuildLog") // domainType
string("Build XCActivityLogParser") // title
string("eyJ0eXBlIjp7ImJsdWVwcmludFByb3ZpZGVyIjp7fX0sImJsdWVwcmludFByb3ZpZGVyX3Byb3ZpZGVyRmlsZVBhdGhTdHJpbmciOiJcL1VzZXJzXC92b3JvYnlvdlwvRG9jdW1lbnRzXC9YQ0FjdGl2aXR5TG9nUGFyc2VyXC9YQ0FjdGl2aXR5TG9nUGFyc2VyLnhjb2RlcHJvaiJ9") // signature
double(711389365.529138) // timeStartedRecording (TimeInterval)
double(63113904000.0) // timeStoppedRecording (TimeInterval)
An array declaration and a description of its elements may look like this:
array(1) // One element array
class_instance(1) // Array element, also of type IDEActivityLogSection
integer(1) // sectionType
string("com.apple.dt.IDE.LogSection") // domainType
string("Prepare packages") // title
string("eyJ0eXBlIjp7ImJsdWVwcmludFByb3ZpZGVyIjp7fX0sImJsdWVwcmludFByb3ZpZGVyX3Byb3ZpZGVyRmlsZVBhdGhTdHJpbmciOiJcL1VzZXJzXC92b3JvYnlvdlwv RG9jdW1lbnRzXC9YQ0FjdGl2aXR5TG9nUGFyc2VyXC9YQ0FjdGl2aXR5TG9nUGFyc2VyLnhjb2RlcHJvaiJ9")
double(711389365.53308) // timeStartedRecording(TimeInterval)
double(711389365.570412) // timeStoppedRecording(TimeInterval)
null // subsections
null // text
null // messages
integer(0) // wasCancelled
integer(0) // isQuiet
integer(0) // wasFetchedFromCache
string("Compile plug-ins and run any prebuild commands") // subtitle
null // location
null // commandDetailDesc
string("52BE500F-D551-461D-975D-BF4B4AA236BF") // uniqueIdentifier
null // localizedResultString
null // xcbuildSignature
And finally, here’s the remaining fields of the root section:
string("") // text
array(0) // messages
integer(0) // wasCancelled
integer(0) // isQuiet
integer(0) // wasFetchedFromCache
null // subtitle
null // location
null // commandDetailDesc
string("2976A337-D8BA-4626-B5F2-41F0F7CB232E") // uniqueIdentifier
null // localizedResultString
null // xcbuildSignature
To help you understand better, here’s the log’s final parsing in JSON format:
{
"version": 10,
"mainSection": {
"sectionType": 0,
"domainType": "Xcode.IDEActivityLogDomainType.BuildLog",
"title": "Build XCActivityLogParser",
"signature": "eyJ0eXBlIjp7ImJsdWVwcmludFByb3ZpZGVyIjp7fX0sImJsdWVwcmludFByb3ZpZGVyX3Byb3ZpZGVyRmlsZVBhdGhTdHJpbmciOiJcL1VzZXJzXC92b3JvYnlvdlwvRG9jdW1lbnRzXC9YQ0FjdGl2aXR5TG9nUGFyc2VyXC9YQ0FjdGl2aXR5TG9nUGFyc2VyLnhjb2RlcHJvaiJ9",
"timeStartedRecording": 711389365.529138,
"timeStoppedRecording": 63113904000.0,
"subSections": [
{
"sectionType": 1,
"domainType": "com.apple.dt.IDE.LogSection",
"title": "Prepare packages",
"signature": "eyJ0eXBlIjp7ImJsdWVwcmludFByb3ZpZGVyIjp7fX0sImJsdWVwcmludFByb3ZpZGVyX3Byb3ZpZGVyRmlsZVBhdGhTdHJpbmciOiJcL1VzZXJzXC92b3JvYnlvdlwvRG9jdW1lbnRzXC9YQ0FjdGl2aXR5TG9nUGFyc2VyXC9YQ0FjdGl2aXR5TG9nUGFyc2VyLnhjb2RlcHJvaiJ9",
"timeStartedRecording": 711389365.53308,
"timeStoppedRecording": 711389365.570412,
"subSections": null,
"text": null,
"messages": null,
"wasCancelled": false,
"isQuiet": false,
"wasFetchedFromCache": false,
"subtitle": "Compile plug-ins and run any prebuild commands",
"location": null,
"commandDetailDesc": null,
"uniqueIdentifier": "52BE500F-D551-461D-975D-BF4B4AA236BF",
"localizedResultString": null,
"xcbuildSignature": null
}
],
"text": "",
"messages": [],
"wasCancelled": false,
"isQuiet": false,
"wasFetchedFromCache": false,
"subtitle": null,
"location": null,
"commandDetailDesc": null,
"uniqueIdentifier": "2976A337-D8BA-4626-B5F2-41F0F7CB232E",
"localizedResultString": null,
"xcbuildSignature": null
}
}
That’s all I wanted to tell you. In the resulting, readable form, you can explore the log and isolate the information that may be useful to your projects.
References
Want to Connect?
Hit me up on LinkedIn, Twitter, or Threads
Let’s parse Xcode logs? was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.