Customers and regulators distrust suppliers because some suppliers (and too many suppliers in that category of "some") behave in fraudulent ways, with outright or systemic fraud taking place too frequently. In that environment, audits - announced or unannounced remain within the "catch them/catch me if you can" sphere. Unannounced audits are designed to "catch them in the act".
The Elitism, Impracticality, Global-Scope Ineffectiveness and Inefficiency of Unannounced Audits:
I find it important to add this point: Unannounced audits are elitist. They are also impractical where they should be most needed given the usual argument in support of such audits. For example, the feasibility of unannounced audits come to mind when considering operations in remote locations in some underdeveloped countries that produce some of the spices, flavoring, thickening, coloring ingredients, etc. Irrespective of rationalizations about why these producers may be exempt, current assessment systems that are deemed to be necessary, never mind unannounced audits, become clearly impractical and elitist. As obviously as can be seen from these arguments, any system that cannot be applied globally to equal degrees is ineffective in safeguarding the global food supply. Unannounced audits fall into this category.
You may provide a comment below if you wish.