How to add a BOOTSEL button to my custom PCB?
Extracted from the Appendix B of the pico datasheet
The "boot" button essentially pulls the QSPI flash memory to ground when pressed. The state of this pin in inspected on power on/boot sequence of the RP2040 to determine whether to boot from flash (HIGH level, button not pressed) or emulate a USB mass storage (LOW level, button pressed).
This approach is different from the dedicated/multiplexed bootstrap pin(s) on a ESP32 or STM32 series' chips
Extracted from the hardware design guide (also linked in the comment on your question) section 2.2
The relevant section goes like so
The QSPI_SS signal is a special case. It is connected to the flash directly, but it also has two resistors connected to it.
The first (R2) is a pull-up to the 3.3V supply. The flash memory requires the chip-select input to be at the same voltage
as its own 3.3V supply pin as the device is powered up, otherwise, it does not function correctly. When the RP2040 is
powered up, its QSPI_SS pin will automatically default to a pull-up, but there is a short period of time during switch-on
where the state of the QSPI_SS pin cannot be guaranteed. The addition of a pull-up resistor ensures that this
requirement will always be satisfied. R2 is marked as DNF (Do Not Fit) on the schematic, as we have found that with
this particular flash device, the external pull-up is unnecessary. However, if a different flash is used, it may become
important to be able to insert a 10kΩ resistor here, so it has been included just in case. The second resistor (R1) is a
1kΩ resistor, connected to a header (J2) labelled 'USB_BOOT'. This is because the QSPI_SS pin is used as a 'boot strap';
RP2040 checks the value of this I/O during the boot sequence, and if it is found to be a logic 0, then RP2040 reverts to
the BOOTSEL mode, where RP2040 presents itself as a USB mass storage device, and code can be copied directly to it.
If we simply place a jumper wire between the pins of J2, we pull QSPI_SS pin to ground, and if the device is then
subsequently reset (e.g. by toggling the RUN pin), RP2040 will restart in BOOTSEL mode instead of attempting to run the
contents of the flash.
How to add usb port to my PCB?
Section 2.4.1 USB of the above hardware design guide covers quite well.
Or, is it possible to program the chip directly used a USB to TTL converter board (over UART)?
Not directly ... unless you write and flash a UART based second stage bootloader (I'm not really sure if this would be that useful as it would very likely much slower than flashing directly over USB).
Unlike chips from STM32* series that support a UART based ROM bootloader, the RP2040 chip doesn't have any on-chip flash but uses a ROM based bootloader (factory programmed and is permanent) to emulate a USB mass storage device to allow drag-n-drop of uf2 files for flashing the external QSPI flash chip. Alternatively you could expose and use the SWD (serial wire debug) pins with a pico-probe (section 4.8 Debugging of the pico-datasheet) for flashing and debugging