We are trying to detect if the AUX (3.5mm) port of the Raspberry Pi is in use so we can automatically swap the output sound to either that or HDMI.
If there is no straight answer, are there any workarounds to this issue?
According to the official schematics of the Pi 3B+ the A/V connector does not feature any mechanisms to detect whether something is plugged in or not. So without additional hardware the Pi cannot know if the jack is "in use" or not.
A circuit to detect if something is plugged in could be connected to the testpins PP25 and PP26... but to cover all possible cases that circuit is not trivial - after all the "something plugged in" might be of high impedance and thus difficult to detect compared to the 1k8 resistors to GND that are already in place. See e.g. here.