Note:
the original version of this article relied heavily on information which hasn’t proved helpful after all. As a result, many of its interpretations appear to have been incorrect, or equally misleading. I pride myself in providing accurate and reliable information, and have therefore rewritten it without such reliance. I apologise unreservedly for my previous errors, and will try to provide increasingly detailed and accurate accounts of this as I get more reliable information in the future.
This account confines itself to the checks which are performed once the XNU kernel has loaded, after the phase conventionally known (inaccurately) as ‘firmware’.
On an M1 Pro Mac which started to boot at around 00 to 01 seconds on the system clock, the first 7-8 seconds of that process is omitted from the log, and consists of the Boot ROM, LLB and iBoot phases. The first log entries from the kernel are
08.289794 === system boot: [UUID]
13.742101 kprintf initialized
with a further gap of more than 5 seconds between them, during which there’s further unrecorded activity. What’s then announced comes from the kernel:
13.742296 Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:41 PST 2021; root:xnu-8019.61.5~1/RELEASE_ARM64_T6000
13.818102 iBoot version: iBoot-7429.61.2
To be able to access LocalPolicy, the kernel needs to load its validation tool for Image4 files, which is does next
13.989331 Darwin Image4 Validator Version 4.2.0: Tue Nov 30 21:45:44 PST 2021; root:AppleImage4-157.60.2~361/AppleImage4/RELEASE_ARM64E
After that, security policy is loaded:
13.989783 AppleMobileFileIntegrity AMFI: UDID enforcement enabled
13.989801 calling mpo_policy_init for AMFI
13.989810 Security policy loaded: Apple Mobile File Integrity (AMFI)
13.989817 calling mpo_policy_init for Sandbox
13.989905 Security policy loaded: Seatbelt sandbox policy (Sandbox)
13.989966 calling mpo_policy_init for Quarantine
13.989970 Security policy loaded: Quarantine policy (Quarantine)
Next comes starting the Secure Enclave key store:
14.065805 AppleSEPKeyStore:319:0: starting (BUILT: Nov 30 2021 21:31:02)
During this period, other parts of the chip are starting up and initialising. There are often copious log entries from each as it does.
So far, all this has been run on a single core. It’s now time to start all the others, which is performed by a series of calls to
cpu_start
, beginning with the first E processor
14.568080 ml_processor_register>cpu_id 0x0 cluster_id 0 cpu_number 0 is type 1
14.568126 cpu_start() cpu: 0
That’s repeated until all ten cores are up and running. On the M1 Pro/Max, these are listed in three clusters:
cluster 0 contains the two E cores, type 1, numbers 0 and 1;
cluster 1 contains the first four P cores, type 2, numbers 2-5;
cluster 2 contains the remaining four P cores, type 2, numbers 6-9.
The original M1 chip has just two clusters:
cluster 0 contains the four E cores, type 1, numbers 0-3;
cluster 1 contains the four P cores, type 2, numbers 4-7.
Gatekeeper is then enabled
14.644687 AppleSystemPolicy GK status: enabled
14.644688 AppleSystemPolicy Per file changetime scans: enabled
The kernel is now ready to access disk storage, so loads the APFS file system
14.779543 apfs apfs_module_start:2568: load: com.apple.filesystems.apfs, v1933.61.1, apfs-1933.61.1, 2021/11/30
The next task is to locate the boot volume, ready to mount its snapshot
14.827815 AppleFileSystemDriver AppleFileSystemDriver: publishing boot-uuid-media=disk3s1 (Macintosh HD)
14.828281 Got boot device = IOService:/AppleARMPE/arm-io/AppleT600xIO/ans@8F400000/AppleASCWrapV4/iop-ans-nub/RTBuddyV2/RTBuddyService/AppleANS3NVMeController/NS_01@1/IOBlockStorageDriver/APPLE SSD AP2048R Media/IOGUIDPartitionScheme/Container@2/AppleAPFSContainerScheme/AppleAPFSMedia/AppleAPFSContainer/Macintosh HD@1
followed by the longstanding announcement of the BSD root
14.828295 BSD root: disk3s1
14.828300 , major 1, minor 13
Only now, nearly 15 seconds after its start, does the Mac get to root from the boot snapshot
14.898802 apfs apfs_vfsop_mount:2118: disk3 Promoter has been locked
14.899252 apfs apfs_vfsop_mount:2188: disk3s1 Rooting from snapshot with xid 183952.
14.899261 apfs apfs_log_mount_unmount:1828: disk3s1 mounting volume Macintosh HD, requested by: kernel_task (pid 0); parent: kernel_task (pid 0)
14.899386 apfs handle_snapshot_mount:885: disk3s1 mounting snapshot w/snap_xid 183952 and sblock oid 0x23263e
The first task with that snapshot is to validate its seal
14.901008 apfs is_root_hash_authentication_required_osx:369: disk3s1 Release kext with internal build: 0, ARV disabled: 0, booting xid: 0
14.901013 apfs is_root_hash_authentication_required:478: disk3s1 root volume, root hash authentication is required
14.901018 apfs authenticate_root_hash:546: disk3s1 successfully validated on-disk root hash
Next comes the Recovery volume within that boot container
14.910181 trying to find and mount BaseSystem dmg as root volume
14.910188 attempting kernel mount for recovery volume...
14.915623 apfs er_state_obj_get_for_recovery:6381: disk3s3 No ER state object for volume Recovery - rolling is not happening, nothing to recover.
14.915679 apfs handle_mount:654: disk3s3 vol-uuid: 4CDE4B2A-F463-49AA-AC77-D4DD774F4832 block size: 4096 block count: 487113810 (unencrypted; flags: 0x1; features: 1.0.2)
14.916518 apfs handle_mount:667: disk3s3 setting dev block size to 4096 from 512
14.916525 apfs nx_volume_group_update:7709: disk3s3 Volume Recovery role 4 Not a System or data volume
14.916803 mounted recovery volume
Other volumes mounted include VM, Preboot and Update
15.095112 apfs apfs_log_mount_unmount:1828: disk3s6 mounting volume VM, requested by: apfs_boot_util (pid 2); parent: launchd (pid 1)
15.097235 apfs apfs_log_mount_unmount:1828: disk3s2 mounting volume Preboot, requested by: apfs_boot_util (pid 2); parent: launchd (pid 1)
15.102171 apfs apfs_log_mount_unmount:1828: disk3s4 mounting volume Update, requested by: apfs_boot_util (pid 2); parent: launchd (pid 1)
Additionally, volumes in the Apple_APFS_ISC container are mounted as required
15.122871 apfs apfs_log_mount_unmount:1828: disk1s2 mounting volume xART, requested by: apfs_boot_util (pid 2); parent: launchd (pid 1)
15.125391 apfs apfs_log_mount_unmount:1828: disk1s1 mounting volume iSCPreboot, requested by: apfs_boot_util (pid 2); parent: launchd (pid 1)
15.128037 apfs apfs_log_mount_unmount:1828: disk1s3 mounting volume Hardware, requested by: apfs_boot_util (pid 2); parent: launchd (pid 1)
The end of the kernel-run phase comes almost 20 seconds after the start. From then on, processes other than the kernel fill much of the log. The first of these loads the Wi-Fi firmware
19.928393 wifiFirmwareLoader wifiFirmwareLoader Sandboxing init issue, couldn't find profile in default paths, attempting default compiled profile
Immediately after that, the system clock is adjusted, and there’s a hiatus when that occurs, here setting the clock back by 3.3 seconds
13.614115 === system wallclock time adjusted
The next task is to validate the Boot Kernel Collection and any auxiliary collection, using
kernelmanagerd
13.642280 kernelmanagerd Starting userland kernel management subsystem (KernelManagement_executables-262.60.4)
14.121399 kernelmanagerd Disabling kext auditing: We are on Apple Silicon
14.137320 kernelmanagerd Initializing with settings:
14.139417 kernelmanagerd finding bundles in repositories:
/Library/Extensions
/Library/Apple/System/Library/Extensions
/AppleInternal/Library/Extensions
/System/AppleInternal/Library/AuxiliaryExtensions
/System/AppleInternal/Diagnostics/AuxiliaryExtensions
/System/Library/AuxiliaryExtensions
/System/Library/DriverExtensions
/Library/DriverExtensions
kernelmanagerd
then validates each of the Boot Kernel Collection with a log entry such as
14.164688 kernelmanagerd validating extension at /System/Library/DriverExtensions/com.apple.DriverKit-AppleUSBFTDI.dext
and many more.
Once that’s ready to load as a collection, that’s reported
14.207370 kernelmanagerd Preparing collection load into kernel
Each is acknowledged as it is loaded
14.316533 kernelmanagerd Received kext load notification: com.apple.kpi.bsd
and many more
Once
kernelmanagerd
has finished, the kernel shuts it down, and no more kernel extensions can be loaded
29.313999 Kext loading now disabled.
29.314006 Kext unloading now disabled.
29.314009 Kext autounloading now disabled.
29.314011 Kernel requests now disabled.
29.314448 kernelmanagerd Kernel requested shutdown. Goodbye!
macOS is now fully loaded and running at last, almost 30 seconds after the start.
Related
Presumably by postcard?
Actually, one of the great joys of the T2 and M1 are that they have stopped marketing from insisting on selling some unfortunate customers Macs with slow internal storage.
Howard.
I don’t know about on an M1 Mac, but on my Intel Mac, booting Big Sur from a USB hard drive takes several minutes. And even after I’m logged in to a working desktop, the drive never seems to stop thrashing, so everything runs too slow for comfort.
The conclusion is that HDDs are not in any way usable with modern versions of macOS. In an emergency, they can let you get access to some critical files, but you really won’t want to be running your Mac like this for longer than the time necessary to install/replace an SSD, (re-)install macOS and restore your documents/apps from a backup.
The problem isn’t macOS, it’s APFS, and I and many others have been saying this since High Sierra. The clincher was Mike Bombich’s careful measurements of fragmentation in the file system metadata.
Please don’t use APFS on hard disks unless (a) the data is going to remain essentially static, as in a disk used as an archive, or (b) it’s a Time Machine backup store, which also remains relatively static in this respect, because of the structure of APFS backups.
Using a hard disk as a boot disk is purgatory.
Howard.
That really is XNU. iBoot2 is a very minimal bootloader and does not log like that. The root filesystem seal cannot be verified by iBoot2 because that doesn’t make any sense; it’s not like the whole OS gets verified, instead it gets verified at runtime dynamically, as XNU reads things from it. What Apple means is that the root hash of the seal is part of the data that is checked by iBoot2 against the Apple tickets and LocalPolicy. That hash is then merely passed to the XNU kernel and it is responsible for checking that the root APFS filesystem is intact and is sealed consistently with it; that is what you’re seeing here.
Every single log line you show is coming from XNU, not iBoot2. iBoot2 does not log in plaintext on production machines (nor does LLB). The iBoot version announced merely comes from the device tree for informational purposes, since iBoot puts it there; iBoot2 is long gone from RAM completely (it gets wiped) by the time of the very first log line here. Darwin does not validate LocalPolicies either (the validator is used for other stuff that uses the same format).
Kexts are managed off-line (which is why you need to reboot to install new kexts) and packaged next to the kernel as an AuxKC loaded by iBoot2. Again by the time the kernel runs none of that needs validating, iBoot2 has already taken care of it. If you look closely, you’ll see what the kernel loaded there is a dext, not a kext; those are different and run in user space, not kernel space, and indeed can be loaded later, as you see here.
Thank you. That confirms my original belief, in which we have always assumed that the Unified Log entries come from the macOS kernel.
However, it doesn’t explain why the boot process repeats the same checks from the kernel when they have only just been performed by iBoot. What’s described in the Platform Security Guide for iBoot is far from being a “very minimal boot loader”, is it?
I’m not disputing what you’ve seen – just that what Apple describes in the guide doesn’t appear to match it. For example, Apple
explicitly states
:
“iBoot is also responsible for verifying the root hash for the signed system volume (SSV) to check that the file system the kernel will mount is fully integrity verified.”
(That’s the last sentence on that page.)
Yet you state that doesn’t make any sense. So, is the Guide wrong – which is a serious accusation – or what?
Howard.
Read the sentence carefully: iBoot verifies the root *hash* (the hash itself, it doesn’t verify the SSV *against* the hash) for the Signed System Volume (SSV, which is not the hash itself) to check that the file system the kernel *will* mount is fully integrity verified (by it, as it mounts and uses it, not by iBoot).
iBoot verifies the kernel and the hash passed to it, and trusts that the kernel (which iBoot knows is a legitimate Apple kernel at that point) will do the appropriate hash checks as data is read from the volume, and trace them to the hash.
You will find these hashes in the Device Tree. Run `ioreg -l -p IODeviceTree | grep auth-blob` to see them. Those are the hashes iBoot2 checks. Not the actual filesystem. iBoot doesn’t even mount or attempt to read the root volume; everything it needs is in the Preboot volume (which is *not* sealed because its contents are signed at the file level, not the filesystem level). If you break the root volume seal without downgrading security, or delete the blessed snapshot outright, you will find yourself with a kernel panic on boot as the *kernel* attempts to mount the filesystem and it fails the check against the hash from iBoot2.
Thank you.
I’m clearly being really stupid here, but I know of no other meaningful method for “verifying” a hash than comparing the recorded hash value against a freshly recomputed hash. If that isn’t performed by iBoot as stated very clearly there, then there has been no verification of the integrity of the SSV. I also consider that you are completely misreading the latter part of that sentence, which states clearly that the verification of the root hash for the SSV is performed as a check by iBoot, and not by the kernel (which is only mentioned in the phrase “the file system the kernel will mount”, i.e. it qualifies which file system will be mounted by the kernel, not what performs the verification).
However, neither of us need waste our time on a detailed parsing or exegesis of that sentence, because the text and diagram clearly don’t represent what actually happens in Monterey. Whether this is because of the age of the document or attempted over-simplification isn’t apparent. However, as a result I have removed all references to information which is outside my immediate knowledge and apologise for my previous errors.
Maybe the next version of the Platform Security Guide will actually correlate with what we now know?
Finally, what you report about a panic occurring with a failed hash check also contradicts what is stated further up the same page: “Any signature verification errors cause the system to boot to recoveryOS to provide repair options.” While a panic might be the result of a bug in Big Sur (in the same way that trying to boot from a removed external disk did), if that has persisted in Monterey, it’s another example of the guide stating what might be aspirations rather than facts.
Howard.
The hash check fail causes a kernel panic, and kernel panic reasons are logged. After multiple panics the system boots to recoveryOS to provide repair options, based on the panic reason. This is an implementation detail. The guide isn’t wrong. You can see this in action if you install two instances of macOS, break the seal on the secondary one, then do a one-time boot from it. The kernel will panic, the machine will reboot into the primary macOS, and you’ll get a kernel panic report from the other OS which will clearly state the SSV failed to verify. This is exactly what happens if you use the Asahi Linux installer, which I developed, and don’t perform the final step that actually installs the Linux bootloader, since at that point the new OS partition is basically a dummy macOS with an empty root filesystem which will fail to verify and mount on boot. I am very familiar with this situation :).
You’re mixing up verifying the *hash* with verifying what is *hashed*. You verify things *against* the hash. When you hash your password and check the hash, you aren’t verifying the hash, what you’re really verifying is the *password*. Sure, the terminology can be a bit confusing, but the intent is clear with a bit of background about how things work. There are long chains of verification here. What Apple is saying is that iBoot is in charge of coming up with a magic number, the SSV root hash, that it has certified as correctly representing the Apple-blessed state of the OS in the root filesystem. The kernel is then in charge of tracing every disk access to this hash via the hash tree on disk.
You will actually find the gory details of the SSV here:
https://support.apple.com/guide/security/signed-system-volume-security-secd698747c9/web
Note how it says that data from the SSV is verified in the *read path*, as data is read from disk. That’s the entire point of a hash tree verification like this (which is also used by e.g. Android with dm-verity): you don’t have to read 15 gigabytes on startup to verify them, instead you verify data as you go. The only thing you have to check on startup is the root of the hash tree. Since data is verified as it is read, that obviously has to be done by the kernel, which is what reads the data. It also further clarifies exactly what validation is happening:
“During macOS installation and update, the seal is recomputed from the file system on-device and that measurement is verified against the measurement which Apple signed. On a Mac with Apple silicon, the bootloader verifies the seal before transferring control to the kernel. On an Intel-based Mac with an Apple T2 Security Chip, the bootloader forwards the measurement and signature to the kernel, which then verifies the seal directly before mounting the root file system. In either case, if the verification fails, the startup process halts and the user is prompted to reinstall macOS.”
So we have an Apple signature, that signs the hash, that then is the root of the SSV hash tree. On Intel Macs, the signature and the hash are passed to the kernel as-is and it verifies the Apple signature, then uses the hash to verify the volume as it is mounted and read. On Apple Silicon Macs, the bootloader verifies the signature over the hash (which involves hashing it again and checking that hash, because that’s how signatures work), and then passes the known-good SSV hash to the kernel, which it then uses to verify the volume as it is mounted and read.
Again, I find no fault in the PSG; of course there are parts that could be clearer, but absolutely nothing I’ve seen so far outright contradicts the reality of the machines (Big Sur, Monterey, doesn’t matter; none of what you covered has significantly changed in Monterey, the changes are around things like Boot Policy update requirements and where recoveryOS is loaded from). The diagram you had was basically fine, and I’m sad you removed all that. The main confusion was around what iBoot2 does and thinking it relates to the kernel logs.
Thank you. I now understand what the guide means when it states:
“iBoot is also responsible for verifying the root hash for the signed system volume (SSV) to check that the file system the kernel will mount is fully integrity verified.”
That sentence should instead read something like:
“iBoot is also responsible for verifying
a saved copy of
the root hash for the signed system volume (SSV)
against its saved hash
, to pass to the kernel so that it can start the verification of the integrity of the SSV when it is mounted later by the kernel.”
I will be revisiting this when I get a moment, to produce clear and accurate descriptions of what does actually happen, rather than the opaque glosses of the guide, and a much improved diagram.
I am extremely grateful for you having the patience and time to get me to realise that iBoot2 doesn’t verify that copy of the hash in the way that almost every other hash verification is performed, against the data from which the hash was originally computed. In a hash tree, that is very quick, of course, as is recorded in the log excerpts above. It’s very unfortunate that Apple’s sole documentation isn’t as explicit as it should be, but assumes the reader already knows that the indirect object of the verb
verify
isn’t the default. That’s why good technical authors are worth their weight in gold.
Howard.
