given that the processor is 64 bits, isn't it obvious that running the OS in 64 bits will be better in every way?
No actually, it's not. In some ways, running a 64 bit operating system could deteriorate the Raspberry Pi's performance.
Benefits of 64 bit:
The two primary benefits of using a 64 bit processor/operating system is that the device can handle more than 4 GB of RAM, and natively handle integers larger than
2^32 without the need for a bignum library.
At the time of writing the Raspberry Pi doesn't have more than 4 GB of RAM (Note: as of Aug. 2020, RPi 4 has from 2 to 8 GB of RAM). At a 1 GB of RAM, you've completely lost the first of the two primary benefits. As for the second benefit, what percentage of people are actually using enough giant numbers that it makes sense for the foundation to support a whole second operating system? As is, the RPi can use huge numbers through software methods, but it seems like if you're going to be consistently in that realm, you need to be using better hardware anyway.
Problems with 64 bit:
The ability to store a larger number isn't granted by magic. Rather, the size of memory objects needs to be increased. In C (and C++) this means changing an
int64_t. This isn't done automatically, hence the comments about the foundation not wanting to maintain two branches.
Additionally, many applications simply don't provide a benefit (for most users) when run in 64 bit mode. Notice that most web browsers, MS Office, and a whole host of other popular software is all still shipped and maintained in a 32 bit manner. Sure you can get your hands on an 64 bit release of MS Office, but it's rarely used.
If the application/operating system is written to take advantage of a 64 bit architecture, your application is going to use more memory, simply because variables and pointers are taking up more space. Usually this is a relatively small trade off for machines that will benefit from the perks. In our case, we have very few perks, and very little RAM.
Also of note:
Just because you're running on a 64 bit machine, doesn't mean the application isn't running as 32 bit. Windows makes this very clear by having two different install paths,
C:\Program Files and
C:\Program Files (x86).
So, will the foundation likely provide 64 bit support?:
We're back at the same point of, "Some people may see benefit, but most will not.". You'll certainly see other projects offering 64 bit builds, but unless the foundation gets a lot of undeserved (imo) flack, they probably won't and shouldn't (imo). Creating and maintaining a separate 64 bit branch isn't a small endeavor, and honestly, just doesn't seem worth it.