The basic problem is that during a DMA operation, peripherals can write into system memory, and the CPU core is not automatically notified.
If any of you guys remember writing drivers or even browsing the exec.library documentation you might remember some functions called
CachePreDMA and CachePostDMA. They essentially marked off certain portions of memory so that they were not in cache, and afterwards, made sure that if the CPU wanted to access data, that it would cache new data from memory and not use stale data in the cache. It's the same principle.
Linux has a different way of doing it; since it has better MMU control it marks off a certain portion of memory as cache-inhibited making it slow to access but guaranteed to be valid (it will never be cached). It will also mark certain regions as being purposed for DMA, in the same way as above (it is simply an API abstraction of a common processor feature).
For the basis of this discussion let us assume that this has ZERO performance impact on the processor or peripherals.
The main problem is the way to run a non-cache-coherent kernel vs. a cache-coherent kernel is completely configured at compile time. This means you can't have a single Efika 5200B/8610/5121E kernel, you need a seperate, specially compiled one for the MPC5121E with the right Kconfig flags set. In this case, though, some advantage can be gained by not having to support every other Power platform at the same time, giving some extra efficiency in the size and modularity.
It should be noted that lack of cache coherency for peripherals is not a problem for userspace which really cannot access the hardware or initiate peripheral access without abstraction anyway (/dev devices etc.) -
so all the same software packages you run on your Linux distro will work fine. It's just a new kernel and new kernel modules you need. This poses (for us) some problem for all the distributions which ship kernel and certain (audio for example) modules in seperate packages. For SuSE, Fedora and Ubuntu this might be hairy. For CruxPPC it is simply another .config file.
Are we all beginning to see what the scope of this is? Basically we need kernel pokers and kernel builders and to create the kernel build systems for the required distributions so we can have installers, kernels and kernel module support for this processor :)