Let's break your question into separate tasks:
- Autonomous orbit determination (autonomous because the DSN won't be there to help you when you need it)
- Autonomous attitude determination
- Situational awareness for formation flying (relative positions, velocities, attitudes and attitude rates), most efficiently done in a cooperative manner
- Cooperative collision avoidance
- Non-cooperative collision avoidance
Let's state it from the start: you can't rely on other craft in the swarm to fulfil your mission since losing one craft implies at best, degraded capabilities for all others, and at worst, mission failure.
Hence, you can't rely on cooperative ways of finding out where the heck your own craft is, and this means extracting range, range rate, attitude, attitude rate information from comms links is a bonus, but never a primary method of running the show. There are some mitigating factors, but they are marginal at best.
Autonomous orbit determination has to contend with uncertainties in:
- asteroid's terrain
- asteroid's gravity
- asteroid's rotation parameters
- all extra perturbations
- systematic biases in the craft's instruments
To overcome all this and more, you've got to use several physical principles (I don't really know the mission you have in mind, so I am not concerned by the exact mass budget) and fuse data from all of them in a smart way (possibly, to uncover systematic errors and compensate for equipment malfunctions):
- passive optical triangulation and ranging (a lot of thought goes into pattern recognition here, luckily you have installed a camera on your craft, haven't you?);
- active optical (ranges, range rates, angles) with a lidar;
- active radar (nice if you can get synthetic aperture pictures from the asteroid, otherwise you are forced to use a terrain model obtained by other means which introduces the possibility of undetected misalignment - your camera and your radar, for instance, are looking at slightly different spots).
For autonomous attitude determination, there's nothing new here: star and Sun trackers, laser gyros, and MEMS accelerometers.
For formation flying and collision avoidance, you have to think about corner cases when one of the sats in the swarm has lost attitude control and is outgassing, threatening to collide with others. You can use optical reflectors and targets to make combined lidar/camera measurements.
If the satellites in your formation have comm links on, you can obtain extra information to feed into your fusion algorithm from the low-level params read from the comms card:
- Doppler frequency shift
- time difference of arrival, including TDOA sent back to your sat by others
Needless to say, run-of-the-mill industrial and retail wireless protocols and cards don't usually cater to this niche segment. You have to design your own protocol and find ways to implement it in hard- (FPGA) and software (SDR).
References:
Grelier et al. Formation flying radio frequency instrument: First flight results from the PRISMA mission. 5th ESA Workshop on Satellite Navigation Technologies and European Workshop on GNSS Signals and Signal Processing (NAVITEC). 2010. DOI (paywalled).
Another look at the PRISMA mission: http://issfd.org/ISSFD_2009/FormationFlyingI/Delpech.pdf
TL;DR
Commercial off-the-shelf wireless won't be able to provide navigation data to swarms operating near an asteroid
Specially designed wireless comms networks will be able to supplement but not replace autonomous navigation and attitude determination equipment
Optical, radar, and wireless comms technologies can and should be used in synergy to improve navigation and formation flying accuracy.