While it may be annoying there is little you can do. Yes turning the wireless ap off for a bit may solve the problem.
Changing the channel and or ssid may help as well but likely only temporary. You should make sure you wpa/2 (with a long random passphrase of 64 characters or as many as your router will allow) in place. Using wep will make it all the easier for an attacker to get into your networks. There are not really tools in BT other than Kismet to see where other AP's in the area are. But without a bit of knowledge and a gps unit this info is not going to help much. Further even if you have this info in hand what are you going to do with it? Are you going to go threaten the person? Call the police? Neither of those are likely help out.
See also http://www.backtrack-linux.org/forum...h-attacks.html