The SharePoint Farm account does get added (and removed) to the Local Administrators group. Ensure you have the following in your XML file.
<Account AddToLocalAdminsDuringSetup="true" LeaveInLocalAdmins="false">
<Account AddToLocalAdminsDuringSetup="true" LeaveInLocalAdmins="false">