As rbrears says, get this sorted NOW - as you are two months in arrears he can start court proceedings BUT you have the mitigation of the mould. There is a Housing Officer attached to your local council; try and get down there today, make an appointment for them to come round asap, and they'll decide whether it's fit for human habitation - and if it's got so bad that you have moss on the carpet, it doesn't sound as if it is.
Have you any record in writing of anything the LL's done to rectify the problem? Did he give you a receipt or any sort of paperwork when he replaced that bit of carpet? Have you written to him, or has it all been done on the telephone so far? I only ask because it would make things a lot easier on you if you have written proof that you asked him to fix things 17 months ago. Don't worry too much if you don't; if it's in a bad state, the Housing Officer can probably judge how long it's been going on for.
How much rent are you paying on the place at the moment? You are entitled to withold part of it as a
LAST RESORT in these sorts of situations; the reason it's a last resort is because of the above, two months in arrears = court actioned eviction - however, you seem to have a genuine reason for doing so and with the Housing Officer's report as evidence you may even be entitled to a partial refund of rent you've paid while the problems were ongoing.
Try very hard not to discuss things over the phone with your landlord now - sending letters makes it much easier to track what everyone's said and promised. You can advise him that, as he has threatened court action, you now wish all communication until the matter is resolved to be in writing.
So, get down the Council, get the Housing Officer down to make a report (play up that you've got a baby in there, they'll be round quicker in theory) and let us know what he/she says. Ask specifically during the appointment about the court action and your
'right of offset' - this is the partial refund you may be entitled to that I mentioned above.
Good luck.
EDIT: Bookie posted while I was typing

You may wish to pay the arrears to get the threat of court action withdrawn, yes - this will take a lot of pressure off you. However, it is worth pursuing the offset and claiming back part of the rent - see what the Housing Officer says. The CAB is the Consumer Advisory Board, you can probably find your local one in the Yellow Pages. They're like this website, but in the flesh with an advisor sitting right there in front of you, going through everything with you.