Currently connected to GPIO Pins as per guide/datasheet and it's not powering up.
Meanwhile I've got another same unit with the same simple setup/circuit except it's powered by 2 AA batteries and it's working fine.
CR2032 cell ratings depend on manufacturers, but in general a typical cell is limited to continuous 1-4mA and peak 6-12mA current draw. The Pico consumption varies widely (between ~1.5mA and ~100mA), but a typical application will need more than 4mA. The wireless module need additional current as well.