Customers and regulators distrust suppliers because some suppliers (and too many suppliers in that category of "some") behave in fraudulent ways. Outright or systemic fraud is rampant. In such an environment, audits (announced or unannounced) remain within the "catch them/catch me if you can" sphere. Unannounced audits are designed to "catch them in the act".
The Elitism, Impracticality, Global-Scope Ineffectiveness and Inefficiency of Unannounced Audits:
I find it important to add this point: Unannounced audits are elitist. They are also impractical where they should be most needed given the usual argument in support of such audits. For example, the feasibility of unannounced audits come to mind when considering small operations that are unable to afford the resources required (funds, personnel and time). That's not all. There is also the feasibility consideration for remote locations in some underdeveloped countries that produce some of the spices, flavoring, thickening, coloring ingredients, etc. How feasible are unannounced audits in such locations? Irrespective of rationalizations about why these producers may be exempt, current assessment systems that are deemed to be necessary, never mind unannounced audits, become clearly impractical and elitist. As obviously as can be seen from these arguments, any system that cannot be applied globally to equal degrees is ineffective in safeguarding the global food supply. Unannounced audits fall into this category.
Another argument that is effectively nullified by the weakest chain link reasoning is that "we can only do what we can, where we can". In other words, we can only strengthen the chain links as much as possible and where possible. Well, you know the rest of the reasoning.
You may provide a comment below if you wish.