I would have to look it up for myself (my memory is crap at the moment ^^"), but one thing to consider is this: if they can fight each other in spirit form, why don't they do so on a regular basis? That implies there's something disadvantageous to doing so even if it is possible to fight in that form.
The thing is, though, that logic applies the opposite way. If staying in spirit mode allows you to avoid a fight, then why not do so? It doesn't seem to take more than a tiny fraction of a second to change form, so you should be able to do it in the time that it takes for a servant to charge a Noble Phantasm. Further, there are plenty of situations in which a servant fights without their master present, and if your logic is right then doing so would be a
massive advantage, since you could just abandon the fight at any time, or even just turn into spirit form and head for the master.
I think that the master issue is probably something to do with it. If one servant stays in spirit form to fight, then the other can just change into normal form and head straight for the master. It's also possible that a servant in normal form can damage one in spiritual form but not the converse, and that fighting in spiritual form would therefore just invite the other servant to change to normal form to defeat you.
Do you ever see Servants in the visual novel fight each other in Spirit Form even when it would be advantageous to do so? Like, say, during the day? They only ever seem to use spirit form to scout and move around to a more tactically sound position.
And I'll remind you in Zero it was pretty much the same thing. Berserkerlot going into spirit form to escape once he got all beat up, Diarmuid going into spirit form to get over the former's jet and then turning physical to stab it with his anti-prana connections spear...
Well, a jet is a physical object, so to interact with that they'd certainly need to be in normal form.