Ok, after a few days of testing and investigation, I have more information to share.
As I mentioned in a previous comment above, the CM3 module connects to the USB hub on the CM3 IO board via the 3rd connector (on the short side). Only the USB OTG is routed to connector U13B (as designated on the schematic here, page 22).
When I look at your schematic, you are connecting the USB hub via the USB switch to the USB OTG pins on the CM3 module. For this to work, you need to do:
echo host > /sys/devices/platform/fe8a0000.usb2-phy/otg_mode
It helps if you set this up as a systemd service to be executed on boot, as described here.
In addition, you have to replace the original DTB with the RP4 DTB:
cd /boot/dtbs/$(uname -r)/rockchip/
sudo cp rk3566-radxa-cm3-io.dtb rk3566-radxa-cm3-io.dtb.orig
sudo cp rk3566-radxa-cm3-rpi-cm4-io.dtb rk3566-radxa-cm3-io.dtb
sudo reboot
However, this is not enough!
I did all of that with my CM3 on the BigTreeTech RPi4B adapter and it still did not work. It seems that the USB hub IC that is selected must also support OTG negotiation upstream!. Thus my RPI4B adapter did not work. But I tried it with this Waveshare CM4 base board, and it works flawlessly. I did not have to do any circuit surgery as described in other posts on this forum. The Waveshare base board uses the VLI VL805-Q6 hub IC.
I also looked where one can source the VLI chip from. The usual suspects (DigiKey, Mouser, Arrow, etc.) do not offer any VLI products, at all. I found it on AliExpress though, for a bit more than 1 USD a piece. So if you need just a few for in-house use, that might be for you. But if you need industrial quantities, then you’ll have to hunt for a proper supplier.
Your other choice would be to look for a different chip, which provides OTG negotiation upstream. I would look up all the manufacturers which products are sold where you buy your electronic components, and look if they have engineering/support forums. You could ask there, if their engineers would recommend a specific IC which satisfies your needs…
Hope that helps - it is certainly possible to have it working (as I can attest right now with my setup), just not with the chip you are currently using.