Whether Apple or Google: Is there a back door into your phone's online backups?

Should you worry about Apple (or an adversary infiltrating its iCloud backup system) being able to unlock your backup for you?