Coparenting With A Meth/alcohol Addict

My husband, a steady alcoholic and meth addict, left two months ago. I was a stay at home mom to four kids and he left us homeless and penniless, moved in with a woman he told me months before was a dealer, and had blocked all contact. I tried and tested to get him to spend time with the kids to no avail. Twice in this two months he has decided He wanted to see just the two year old. The last time was on Halloween. When I took her to see him, along with our five month old, He first dropped the five month old (He was ok thank God).then the woman he moved in with showed up even tho he said she wouldn't be there, and when I told him the toddler hadn't been trick or treating yet he started screaming at me then the woman jumped in my face screaming ash me and threatening me while I stood there calmly holding my infant. I tried to remove the kids from the situation and was physically prevented from doing so. I went ahead and left the toddler there as I felt I had no choice. My husband then called me three hours early and demanded I come get her immediately because I had ruined his evening. When I got there, the girlfriend screamed obscenities at me the whole time in front of the kids, they'd driven around without even securing the toddlers car seat into the car, and she hadn't eaten even tho it was 8 pm by the time I got there. I told my husband at that time that he had showed he was incapable of providing a safe environment, and he appeared to be high, so until we get a custody agreement I told him He's welcome to come see them every day if he wants but that he can't take them anywhere or be alone with them. Since then he's had no contact at all. He apparently sobered up for a few days, long enough to pass a drug test at his new job and two realize he missed his kids, and he called and informed me that he's taking the toddler for the weekend. Didn't ask. Didn't discuss. Just told me. I told him He's welcome to come see them and then hung up. He's making himself out to be a victim, telling everyone I'm keeping his kids from him, when in reality he just left and has chosen not to see them or talk to them or support them. I know there will be an explosion this weekend when I refuse to comply with his bullying or put my kids in harm's way. He claims he's clean but his appearance and behavior say otherwise. And I know that having his daughter is not enough to keep him from using. If it was, I'd not have found pipes and baggies where the kids could easily have gotten them in the weeks before he left. I also found messages that indicated he was dealing before he left.
This is all so hard. I don't even know if I'm making sense. Is anyone else trying to co parent with an addict? Found yourself abandoned with no notice? Have advice? Encouragement? I know he won't be getting clean any time soon, his mother is letting him stay with her and in denial and trying to make me the bad guy.
Honestly, I think you're going to have to go through the court system, or even child protective services if you feel that the environment over there is bad for your toddler.
There's not much else you can do other than not letting the kids go there. If he really wants to see the kids that bad let him take you to court, and when you're there tell them why you won't let them go. Insist on a drug test as part of his visitation.

If he does show up demanding to take the kids, call the police and let them sort it out.