Of course, even if the published source code of a mainstream project is audited, there still remains the issue that in reality, all but the ultranerdiest of übergeeks will actually bother to obtain the software by compiling it from source. Most people will simply download the ready-compiled binaries without taking the time or having the expertise to check whether the binaries they are merrily installing on their computer actually match the published source code.
Recently, master's student Xavier de Carné de Carnavalet decided to attempt such a feat for popular file encryption tool TrueCrypt. His report demonstrates that checking that a published binary actually matches the published source code for a given project is something of a labour of love, and that various assumptions need to be made. Luckily in the case of TrueCrypt, the result does give us a level of confidence that the binary does not hide any malicious code (assuming none is hidden in the published source code). What is slightly alarming is that this type of basic security check is far from being a routine process that any user or even developer could accomplish.
TrueCrypt has come under the spotlight because its authors are an anonymous foundation and it is an obvious target for backdoors on the part of hackers or intelligence agencies. But there is surely much more mainstream software (such as free, open source antivirus programs) that should be given such attention.
As far as an audit of the TrueCrypt source is concerned, at least earlier versions have been subject to a basic security audit, but apparently from the point of view of detecting accidental security flaws in the source code rather than deliberate backdoors or vulnerabilities. This is why cryptography Matthew Green, for example, has recently called for a public, properly funded audit of TrueCrypt. In the light of recent revelations on how intelligence agencies have been compromising mainstream software such as antiviruses, I wonder how many more popular open source projects should be subject to the same scrutiny and haven't been.