Change your Boot Mode in Debug Settings to System Software. Also, make sure NP Environment is set to NP.
Managed to fix it. Tried out a debug with non-host mods and the RTM tools finally decided to connect just fine. Thanks for the help though !
One last question. When trying to make a debug eboot, I've seen several videos saying different things. Do I take the original EBOOT and convert it that way, or do I take the mp_default.self and convert it to an EBOOT that way ? I tried the first way but that didn't seem to make a functional debug eboot xD.