OS-6964: Cloud Firewall not affecting bhyve Instances

Details

Issue Type:Bug
Priority:4 - Normal
Status:Resolved
Created at:2018-05-16T21:09:41.606Z
Updated at:2018-10-24T18:44:32.917Z

People

Created by:Ryan Kitchen
Reported by:Ryan Kitchen
Assigned to:Former user

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2018-10-19T19:19:03.349Z)

Fix Versions

2018-10-25 Dot Com (Release Date: 2018-10-25)

Labels

bhyve

Description

Enabling the cloud firewall does not appear to affect Bhyve instances. More details can be found here: https://jira.joyent.us/browse/QA-262

When the firewall is enabled an LX host blocks traffic as expected, and honors additonal rules.
When the firewall is enabled on bhyve the default rules do not take effect, and rules to block specific traffic also do not block it.

Comments

Comment by Former user
Created at 2018-05-16T22:03:38.035Z
Updated at 2018-05-16T22:04:52.700Z

In order for bhyve to hook into cloud firewall we need to implement firewall hook support into viona; much like was done for KVM/VND. We can probably use vnd_hook() as a reference implementation – though the details may differ. I'm not sure how long this will take but it's my opinion that the "smart checksum" and LSO work is higher priority as it should benefit both inter and intra-host network performance greatly, and the user can also use iptables inside the guest until we implement cloud firewall support.


Comment by Former user
Created at 2018-09-14T05:15:38.484Z

I have a working solution, updated to the viona-ipf branch on GitHub. The consensus is that it will be more beneficial to wait for OS-2340, OS-6778, OS-6794, and OS-6884 to integrate (all are currently under review) before submitting this code for review (a minor amount of tweaking will be required after that, but should be fairly minimal) because of the robustness improvements to the viona driver they include.


Comment by Former user
Created at 2018-09-26T21:02:23.596Z

For testing (this was all in a SmartOS instance running under Vmware fusion), I created two bhyve instances running 56b9596c-a3bf-11e8-b5c9-0f79b1e762c5 (centos-7@20180819) whose network interfaces were created on an etherstub.

I started with a default set of rules:

[root@viona ~]# fwadm list
UUID                                 ENABLED RULE
2bad8e10-4be9-439c-ab53-639f687f3563 true    FROM any TO all vms ALLOW icmp TYPE all
5cefcd83-f170-4799-8601-1c9c9c9e2d40 true    FROM any TO all vms ALLOW tcp PORT 22
c22c3437-09ee-46bf-a989-6dfc0fb70c4b true    FROM any TO all vms ALLOW icmp6 TYPE all

I then started pings from each VM to the other VM and verified that both pings were successful. I then disabled the ICMP rule via fwadm disable 2bad8e10-4be9-439c-ab53-639f687f3563. I then verified that the pings stopped between the instances. I then re-enabled the rule, and saw that the pings then resumed.

I then tried to scp a 2.2Mb file (vim) from one instance to the other with 5cefcd83-f170-4799-8601-1c9c9c9e2d40 enabled, which was successful. I then disabled 5cefcd83-f170-4799-8601-1c9c9c9e2d40 and tried to scp again (which failed with a connection timeout after a while).
I then re-enabled the rule and tried it again (which succeeded).


Comment by Former user
Created at 2018-10-17T22:20:09.015Z
Updated at 2018-10-17T22:20:52.783Z

We've also setup a test rig w/ two CentOS VMs (both were given a tag of 'test').
They were created with cloud firewall disabled.

Running iperf3:

[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  2.52 GBytes  2.16 Gbits/sec  552             sender
[  4]   0.00-10.00  sec  2.51 GBytes  2.16 Gbits/sec                  receiver

Once cloud firewall was enabled, my existing ssh sessions to the two VMs died and attempting to run iperf3 failed (as expected). Then rules for SSH as well as iperf3 were added:

fwadm add -g -e 'FROM any TO all vms ALLOW tcp PORT 22'
fwadm add -g -e 'FROM any to tag "test" ALLOW tcp PORT 5201'
fwadm add -g -e 'FROM any to tag "test" ALLOW udp PORT 5201'

And ssh connections were again successful, and iperf3 was able to run (note this run was for longer -- 60s):

[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  13.9 GBytes  1.99 Gbits/sec  5764             sender
[  4]   0.00-60.00  sec  13.9 GBytes  1.99 Gbits/sec                  receiver

Comment by Former user
Created at 2018-10-18T19:28:19.368Z

Letting iperf3 run for a bit over two hours (restarting after every 10s run) w/ the above rules in place:

Average 1.79 Gbits/s Stddev=0.16 Time=8220s (137m)

Comment by Jira Bot
Created at 2018-10-19T19:19:06.691Z

illumos-joyent commit 504b3e4c2ecff2fc5ce62559b4ba11c33e33027c (branch master, by Jason King)

OS-6964 Cloud Firewall not affecting bhyve Instances
Reviewed by: Ryan Zezeski <rpz@joyent.com>
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Approved by: Patrick Mooney <patrick.mooney@joyent.com>