if you don't want X, why load it? stay at the command-line all you want.
that being said, if you really wanted to remove X/KDE/Flux/ etc, as well as all their libraries and dependencies... then yes there is certainly a way. this isn't windows, after all.
removing it is easy (#man apt) but doing right will take a full-blown project.
good luck with that...(just don't expect a lot of assistance from these boards, you're kinda on your own with this.)