The answer is “the answer when only rows can be swapped” multiplied by “the answer when only columns can be swapped.” This can be seen from the fact that swapping rows does not affect the condition for swapping columns.

Let us find “the answer when only rows can be swapped.”

If we draw an edge between row vectors that can be swapped, it turns out that we can swap connected vectors in any way we like.

For example, if we want to swap A and D in A - B - C - D, we can do so as follows:

A - B - C - D
-> B - A - C - D
-> B - C - A - D
-> B - C - D - A
-> B - D - C - A
-> D - B - C - A

Thus, the answer is the product of (the size of a connected component)! over all connected components. You can, for example, use atcoder/dsu to implement this solution.