Open in app

Sign in

Write

Sign in

Troubleshooting tales — How to connect two QEMU guests via a bridge interface

Paulo Almeida
9 min readFeb 2, 2024

--

Recently, I wanted to study how DRBD worked closely and to accomplish it I always thought that being able to set a break-point and walk-through the routines to be the fastest way of learning anything.

DRBD runs mostly in kernel-space which left me with no alternatives but to run 2 QEMU guests — one as the primary and another as the secondary resources — both attached to a GDB session for debugging…. What could go wrong, right?

high-level representation of DRBD architecture

Network set up

Conceptual topology of how these 2 guests would communicate with each other

Creating Bridge — Host Level

sudo ip link add br0 type bridge
sudo ip addr add 192.168.0.1/24 dev br0
sudo ip link set br0 up

Launching QEMU guests — Host Level

PS: I won’t add the kernel/GDB parameters in here — makes it easier for readers

# Guest 1
qemu-system-x86_64 \
        -M pc \
        -kernel ~/workspace/linux/build/arch/x86_64/boot/bzImage \
        -initrd ~/workspace/minimal_linux_fs/images/rootfs.cpio \
        -append "console=tty1 console=ttyS0"  \
        -net nic,model=virtio -net user \
        -netdev bridge,id=hn1 \
        # note that MAC addr is different
        -device virtio-net,netdev=hn1,mac=e6:c8:ff:09:76:99 \
        -m 1024 -d guest_errors -nographic

# Guest 2
qemu-system-x86_64 \
        -M pc \
        -kernel ~/workspace/linux/build/arch/x86_64/boot/bzImage \
        -initrd ~/workspace/minimal_linux_fs/images/rootfs.cpio \
        -append "console=tty1 console=ttyS0"  \
        -net nic,model=virtio -net user \
        -netdev bridge,id=hn1 \
        # note that MAC addr is different
        -device virtio-net,netdev=hn1,mac=e6:c8:ff:09:76:9c \
        -m 1024 -d guest_errors -nographic

Configuring QEMU guest interfaces — Guest Level

# Guest 1
ip addr add 192.168.0.8/24 dev eth1
ip link set eth1 ip

# Guest 2
ip addr add 192.168.0.16/24 dev eth1
ip link set eth1 ip

Checking if everything is working as expected

On the host-level I check if I can access both 192.168.0.8 and 192.168.0.16 — all seems to be in order here

[paulo@fedora ~]$ ping -W 1 -c 1 192.168.0.8
PING 192.168.0.8 (192.168.0.8) 56(84) bytes of data.
64 bytes from 192.168.0.8: icmp_seq=1 ttl=64 time=0.776 ms

--- 192.168.0.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.776/0.776/0.776/0.000 ms

[paulo@fedora ~]$ ping -W 1 -c 1 192.168.0.16
PING 192.168.0.16 (192.168.0.16) 56(84) bytes of data.
64 bytes from 192.168.0.16: icmp_seq=1 ttl=64 time=1.40 ms

--- 192.168.0.16 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.396/1.396/1.396/0.000 ms

On the guest-level something looks funny. I can ping the gateway (bridge) 192.168.0.1 but I can’t ping the other guest.

# Guest 1
ping -W 1 -c 1 192.168.0.16
PING 192.168.0.16 (192.168.0.16): 56 data bytes

--- 192.168.0.16 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

# Guest 2
ping -W 1 -c 1 192.168.0.8
PING 192.168.0.8 (192.168.0.8): 56 data bytes

--- 192.168.0.8 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

Surely this must be some iptables/nftables configuration, right? So let’s check what info we can use to troubleshoot this

wireshark

wireshark doesn’t say much apart from the fact that no response was found for the ping request

nftables

nftable caught my attention because the forward policy is set to drop by default — add that to the fact that we have no rules for our bridge br0 and you might have a solution, right?

# Zero counters
[root@fedora paulo] iptables -Z

# Checking iptables/nftables rules
[root@fedora paulo] nft list table filter

table ip filter {
 chain DOCKER {
 }

 chain DOCKER-ISOLATION-STAGE-1 {
  iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
  counter packets 0 bytes 0 return
 }

 chain DOCKER-ISOLATION-STAGE-2 {
  oifname "docker0" counter packets 0 bytes 0 drop
  counter packets 0 bytes 0 return
 }

 chain FORWARD {
  # That looks funny... (policy drop)
  type filter hook forward priority filter; policy drop;
  counter packets 0 bytes 0 jump DOCKER-USER
  counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-1
  oifname "docker0" ct state related,established counter packets 0 bytes 0 accept
  oifname "docker0" counter packets 0 bytes 0 jump DOCKER
  iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept
  iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
 }

 chain DOCKER-USER {
  counter packets 0 bytes 0 return
 }

 chain INPUT {
  type filter hook input priority filter; policy accept;
 }

 chain OUTPUT {
  type filter hook output priority filter; policy accept;
 }
}

For now I’m only using ICMP so this simple rule should suffice

# add rule for interface br0
[root@fedora paulo] nft add rule ip filter FORWARD \
                      iifname "br0" oifname "br0" accept

# Checking iptables/nftables rules
[root@fedora paulo] nft list table filter

table ip filter {
 chain DOCKER {
 }

 chain DOCKER-ISOLATION-STAGE-1 {
  iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
  counter packets 0 bytes 0 return
 }

 chain DOCKER-ISOLATION-STAGE-2 {
  oifname "docker0" counter packets 0 bytes 0 drop
  counter packets 0 bytes 0 return
 }

 chain FORWARD {
  type filter hook forward priority filter; policy drop;
  counter packets 0 bytes 0 jump DOCKER-USER
  counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-1
  oifname "docker0" ct state related,established counter packets 0 bytes 0 accept
  oifname "docker0" counter packets 0 bytes 0 jump DOCKER
  iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept
  iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
  # There she is! 
  iifname "br0" oifname "br0" accept
 }

 chain DOCKER-USER {
  counter packets 0 bytes 0 return
 }

 chain INPUT {
  type filter hook input priority filter; policy accept;
 }

 chain OUTPUT {
  type filter hook output priority filter; policy accept;
 }
}

Let’s try that again

For some reason that didn’t do the trick…

# Guest 1
ping -W 1 -c 1 192.168.0.16
PING 192.168.0.16 (192.168.0.16): 56 data bytes

--- 192.168.0.16 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

# Guest 2
ping -W 1 -c 1 192.168.0.8
PING 192.168.0.8 (192.168.0.8): 56 data bytes

--- 192.168.0.8 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

I wondered if the nftables rule was even hit in the first place. That’s when things became very confusing to me.

3 things happened and they can not all be true at the same time

  1. the rule #7 of the FORWARD chain shows that both pkts and bytes are set to 0 — so it wasn’t hit
  2. the default policy of the chain counter shows DROP 0 packets, 0 bytes — so it wasn’t dropped
  3. targets DOCKER-USER and DOCKER-ISOLATION-STAGE-1 seem to have hit the RETURN action — so it couldn’t have “dropped” the packets
[root@fedora paulo]# iptables -L -v -n --line-numbers 
Chain INPUT (policy ACCEPT 26 packets, 10784 bytes)
num   pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy DROP 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1        1    84 DOCKER-USER  0    --  *      *       0.0.0.0/0            0.0.0.0/0           
2        1    84 DOCKER-ISOLATION-STAGE-1  0    --  *      *       0.0.0.0/0            0.0.0.0/0           
3        0     0 ACCEPT     0    --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
4        0     0 DOCKER     0    --  *      docker0  0.0.0.0/0            0.0.0.0/0           
5        0     0 ACCEPT     0    --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
6        0     0 ACCEPT     0    --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           
7        0     0 ACCEPT     0    --  br0    br0     0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 3 packets, 324 bytes)
num   pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (1 references)
num   pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 DOCKER-ISOLATION-STAGE-2  0    --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
2        1    84 RETURN     0    --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 DROP       0    --  *      docker0  0.0.0.0/0            0.0.0.0/0           
2        0     0 RETURN     0    --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1        1    84 RETURN     0    --  *      *       0.0.0.0/0            0.0.0.0/0

Looking at wireshark confirmed to me that there was something else blocking the packet — note that now there is a Destination unreachable packet from 192.168.0.1 (br0) to 192.168.0.16 (Guest VM 2)

My only card up my sleeve was to take a look at the program that configure that nftables/iptables rules in the kernel — that is a program called firewalld

# Stop firewalld - which despite what people think, it doesn't stop 
# the "firewall" as the rule already configured in the kernel remain so
# by default
systemctl stop firewalld.service 

# Running firewalld with logging verbosity
firewalld --nofork --debug 1

While I couldn’t find any log entries that could indicate why my ICMP packets where being dropped, I saw something that caught my attention

2024-02-03 10:37:09 DEBUG1: zone.getInterfaces('docker')
2024-02-03 10:37:09 DEBUG1: zone.addInterface('docker', 'docker0')
2024-02-03 10:37:09 DEBUG1: Setting zone of interface 'docker0' to 'docker'
2024-02-03 10:37:09 DEBUG1: Applying policy (zone_docker_HOST) derived from zone 'docker'
2024-02-03 10:37:09 DEBUG1: Applying policy (zone_HOST_docker) derived from zone 'docker'
2024-02-03 10:37:09 DEBUG1: Applying policy (zone_ANY_docker) derived from zone 'docker'
2024-02-03 10:37:09 DEBUG1: Applying policy (zone_docker_ANY) derived from zone 'docker'
2024-02-03 10:37:09 DEBUG1: zone.InterfaceAdded('docker', 'docker0')

The reason why this is interesting is because docker0 is also a bridge interface and after some quick tests I could verify that ICMP packets from docker containers going through that bridge worked just fine.

That being said, the only difference is that docker0 is part of a zone called docker and that zone has forward: yes

# List all zones
[root@fedora paulo] firewall-cmd --get-zones
block dmz drop external home internal libvirt public trusted work

# List all active zones
[root@fedora paulo] firewall-cmd --get-active-zones
FedoraWorkstation (default)
  interfaces: wlo1
docker
  interfaces: docker0

# Inspect docker zone
[root@fedora paulo] firewall-cmd --zone=docker --list-all 
docker (active)
  target: ACCEPT
  ingress-priority: 0
  egress-priority: 0
  icmp-block-inversion: no
  interfaces: docker0
  sources: 
  services: 
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

My options are:

  • Create a new zone for DRBD
  • Make use of one of the inactive zones that have forward: yes

The zone called trusted seem to fit the criteria as it has pretty much the same config that docker zone has

[root@fedora paulo] firewall-cmd --zone=trusted --list-all 
trusted
  target: ACCEPT
  ingress-priority: 0
  egress-priority: 0
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: 
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

Add “br0” to “trusted” zone

# Add interface
[root@fedora paulo] firewall-cmd --zone=trusted --add-interface=br0
success

# List active zones
[root@fedora paulo] firewall-cmd --get-active-zones
FedoraWorkstation (default)
  interfaces: wlo1
docker
  interfaces: docker0
trusted
  interfaces: br0 # There she is!!!

Testing everything (again)

# Guest 1
ping -W 1 -c 1 192.168.0.16
PING 192.168.0.16 (192.168.0.16): 56 data bytes
64 bytes from 192.168.0.16: seq=0 ttl=64 time=1.386 ms

--- 192.168.0.16 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.386/1.386/1.386 ms

# Guest 2
ping -W 1 -c 1 192.168.0.8
PING 192.168.0.8 (192.168.0.8): 56 data bytes
64 bytes from 192.168.0.8: seq=0 ttl=64 time=1.362 ms

--- 192.168.0.8 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.362/1.362/1.362 ms

Conclusion

That was tough one to debug. To be honest it took me much longer than I care to admit.

Clearly, there is something that could be changed either at the nft, iptables or even at the kernel log messages to indicate that something else was also acting on the packet filtering. As is, it was pretty misleading for me — Most likely for someone with a network engineering background that is “obvious” in hindsight but this is probably the exemption rather than the rule.

Hope that this has helped you get your QEMU guests communicating ;)

Acknowledgment

While my blog post was mostly focused on the troubleshooting part, the actual network topology was gotten from this blog post. In his case he didn’t face the iptables/zone issues — mostly likely due to his distro default settings. I see our blog posts being complementary.

Kernel
Qemu
Virtualization
Linux
Network

--

--

Paulo Almeida
Paulo Almeida

Written by Paulo Almeida

81 Followers
10 Following

Interested in technical deep dives and the Linux kernel; Opinions are my own;

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams