On a off-line Pi configured as a hotspot thanks to Using your new Raspberry Pi 3 as a Wi-Fi access point with hostapd, I configured a hard coded 40 IP address limit. Seems to handle all of them properly. Because using web sockets for real-time interactions among all connected users on my Node.js app, I didn't try more.
More generally, I think that though the Pi may offer this feature with a bit of custom configuration, it is not optimized for this use. If you wan't to get satisfying Wi-Fi performance for each client, I suggest you restrain the number of physical clients: I mean, it's not because it's wireless that each client won't consume less resource about handling its connection / IP session.
Many routers offer to assign much more IP addresses, but that's about DHCP and assigning addresses to connected clients, not about handling 50 Wi-Fi traffic-wise distinct connections and contexts.
Ex: All Apple Airport stations have a limitations of 50 users for instance... see Compare AirPort family.
Months (!) later, eventually had the opportunity to test with lots of devices. It appears that the Pi 3B with a Jessie based Raspbian and its default WIFI component could accept no more than 32 MAC addresses, even with dnsmasq configured to allow up to 60 IPs.
Based on some Raspi Community Forums thread I assume this might be caused by a hard coded limit in the brcmfmac driver for the WIFI chip used on the board, still looking into this...
The Pi 3B+ with Stretch based Raspbian (9.4) could accept only up to 16 connections (same hostapd/dnsmasq config). As this is only half as much as the Pi 3B, I have nothing but hypothesis for this:
-either the Stretch embedded brcmfmac driver for the WIFI chip now limits to 16 MAC addresses (unlikely IMHO)
-or it is linked to the new WIFI chip used in the new board, now dual band 2,4/5 GHz when in client mode, but I could configure the AP only using the 2,4GHz, so only half the resources would be allocated therefore half the Pi 3B maximum capacity as a result (seems more plausible to me, but definitely not verified)