I'm planning to make a VMWARE ESX virutal server running 4 XP 'machines' with 2 modems. Is it possible to separate this modem card in VMware or does it act as one modem?

You most likely CANNOT split the same cards with 2 modems each across multiple virtual servers. When the driver is loaded, it will take control of the entire card, only one virtual server can own it.