I decided to answer myown Q because I dug through a lot of material before asking here, and found a lot of solutions dealing with similar, but not the same issue.
Thanks to @Andyroo the why this happened became clear.
So, newest Raspberry Pi OS changed policy regarding the analog 3.5 jack audio output, and the HDMI.
Now, the 3.5 output is registering to ALSA as a different audio card.
Now you will have card 0 = HDMI 1 & card 1 = Headphones. I have no idea what would happen if you have HDMI 2 connected on Pi 4, did not try it yet. I think audio output is not supported on HDMI 2, but I am not completely sure.
The old way was to have HDMI and 3.5 output registered as subdevices of the card 0.
For some reason, when you upgrade Raspbian Buster through apt, you will get the new policy adopted. The new way is very good and it should have been implemented from the start. But nasty surprises like this shouldn't be happening.
What should also happen, when you got the newest ALSA driver and settings is also that, if you have selected 3.5 as your default output, the card 1 should be selected as your default output automatically.
Instead your audio will be routed back through HDMI and you will be left wandering about what is going on.
Also, the updated raspi-config should properly configure the output under the advance settings, but it does not. It does it the old way, and, naturally, nothing changes.
The solution is to:
Add the following lines, either to your /etc/asound.conf file (global configuration) or to your ~/.asoundrc file (per session configuration), as you prefer/need.
defaults.pcm.card 1
defaults.ctl.card 1
if you already made similar settings, change them or add them to the config file. If the file does not exist, create it.
This solves your ALSA problem. However, if you use pulseaudio this will do nothing to change its settings, as it does not inherit its settings from ALSA, and pulseaudio's output will be going through HDMI.
To solve this you have to add/change the following line in /etc/pulse/default.pa script, or, if you use system wide configuration, /etc/pulse/system.pa script.
set-default-sink 1
It is better practice to name the sink instead using its index, as the OS can decide to change it from boot to boot, especially if you add a new card, but Raspbian does not do that, for now, so this will work nicely. 0 is HDMI, 1 is Headphones, and external cards you should add by their name if you use more of them as their enumeration is unpredictable.
You also have to set the default sink to 1 (or name it) in /etc/pulse/client.conf.
I do not use the jack server, so I do not know how it will react to new changes and whether it inherits ALSA settings or not. So I cannot help jack's users directly, but, I think information about these changes will be enough to help. If someone ends up with this problem, please post a solution in a comment to make someone's job faster and easier.
