At the moment I am discovering a bug in the Linux Kernel.
The scenario is as follows:
There is a router using virtual interfaces (e.g. VLANs) eth1.100 and eth1.200.
Those interfaces share a MAC address. In IPv6 there are so called "link-local" addresses.
These addresses are build from an static prefix + the MAC address.
On the router runs Quagga with its ospf6d (doing OSPFv3) configured with eth1.100 and eth1.200 in area 0. Quagga starts to send OSPF hello packets on both links. On eth1.100 it sends the packets with an interface ID of 4. On eth1.200 is sends the packets with an interface ID of 5.
As I've discovered in the Quagga source code, the packets are bound to an interface by using the struct in6_addr with its property sin6_scope_id.
So far, everything is ok.
When I now activate IPsec in transport mode with AH only (have not tested other setups yet) for protocol 89 (ospf) all OSPF packets have an AH added. But: The packets seem to loose their associated interfaces and are being sent on all interfaces. The problem reside somewhere in the XFRM stack of the Linux Kernel.
In the mean time I found out that
1. the problem does not exist in kernel version 3.4 but in 2.6.32
2. the problem resides in net/xfrm/xfrm_policy.c in the __xfrm_policy function where it calls xfrm_find_bundle(fl, policy, family).
xfrm_find_bundle returns a struct dst_entry. dst_entry is a linkes list and the last element is being used for routing. xfrm_find_bundle gets the fl parameter where fl.oif is the number of the outgoing interface that should be used. However that interfaces seems to be ignored in some cases, so the dev structure in dst_entry does not equal the fl.oif interface.
I don't think I'll invest more energy on this bug as it does not exist in 3.4 anymore.
However, here is a hotfix for the problem (in case you have this problem and can not easily upgrade your kernel):
diff linux-184.108.40.206/net/xfrm/xfrm_policy.c linux-220.127.116.11.orig/net/xfrm/xfrm_policy.c
< /* HACK: get sure that packet is being transmitted out of fl->oif! */
< if(dst != NULL)
< dst->path->dev = dev_get_by_index(net, fl->oif);