You can build up your Virtual Private Cloud (VPC) on top of the public cloud using Zenlayer bare metal.
Last updated
You will get the full horsepower of the server, and you also have full access to the hardware layer, e.g., BIOS/network adapter driver.
To build a VPC over the Zenlayer bare metal instances, we will perform the following steps:
Spin up a bare metal instance and get IP configured. After the initial provisioning, your instance is assigned with a /30 internet IP address, and this is how it works.
For example, the CIDR notation 128.14.148.248/30 has 4 IP addresses included, and 2 out of these 4 are usable for hosts. The server will take one while the network switch will take another one as gateway IP:
128.14.148.248 - Network identifier (not used for server)
128.14.148.249 - Gateway
128.14.148.250 - Server’s internet IP address
128.14.148.251 - Broadcast address
Assume that a user chooses the ubuntu 20.04 as the OS and here it is the ip a output from console:
To build a VPC, the plan is to get one server for the private cloud controller and two servers for the virtual machines, therefore placing the order of 3 bare metal instances and got them running:
After the initial provisioning, 3 instances are up and running with Internet access. As those 3 instances are still isolated from each other, the next step is to build a private network and bridge all of them. Zenlayer bare metal cloud provides VPC, that is a VLAN/VxLAN, which can help you to achieve this.
Click Add Instance to attach instances to your subnet.
You can get the MAC address of VPC on instance details page, which you may use it for future setup with DHCP IP & MAC binding.
Note
The page shows pxe 0 as Zenlayer uses that same interface for OS installation during the provision when you placed the order, and now they are your private LAN interface.
You can see the VLAN ID is 1047 in this case. All the Zenlayer bare metal instances' switch ports are set to “access port” so this VLAN ID is for reference purpose in case you are requesting support and may be asked to provide info.
In this case, the hostname and IP address assignment are shown as below:
Test802Q-01
10.7.7.11
0c:c4:7a:e2:df:42
Test802Q-02
10.7.7.12
0c:c4:7a:e2:db:1a
Test802Q-03
10.7.7.13
0c:c4:7a:e2:e1:a2
With the private network IP address assigned you can ping to each instance to verify the private network is up and running:
To build your own private cloud, a controller server is the key module by making VM commission/deploy/destroy automatic. There are many options vary from commercial software to open-source software, and in this case, MAAS will be taken as an example to build a KVM-based private cloud. The private cloud topology is as below:
Test802Q-03 server - Controller server with MAAS (control plane)
Test802Q-02 server - KVM Server 2 (data plane)
Test802Q-01 server - KVM server 1 (data plane)
See MAAS | How to install MAAS to install the MAAS on the Test802Q-03 instance.
Access the instance via https://[$server IP]:5240/MAAS/r/dashboard. For security reason, block the Internet access of MAAS GUI port, and make an SSH tunnel port forwarding so that access to the GUI with the URL http://127.0.0.1:5240/MAAS/r/dashboard is allowed.
Enable DHCP service
Go to Subnets tab and enable DHCP service on the LAN interface (fabric-0 in this case). Check MAAS provides DHCP and set the IP range accordingly ( 10.7.7.x/24 in this case).
Go back to Controllers tab to double confirm DHCP is up and running.
MAAS user configuration and upload your SSH public key
With the completion of MAAS setup, a user “maas“ was created:
The output would be like: “#: maas:x:117:124::/var/lib/maas:/usr/sbin/nologin”, showing the maas account doesn’t have login shell by default. Instead for managing the KVM/Hypervisor servers, we need to enable the bash/shell for it and create the SSH key for login to the KVM/Hypervisor servers:
This will enable the bash and generate RSA key pairs in .ssh subfolder of maas account’s home directory.
We will copy the content of id_rsa.pub to KVM servers so that maas account can log in to KVM/Hypervisor servers with key authentication.
You will be asked to upload another key for deploying servers by MAAS, as there is no default username/pass combo MAAS deploys machines with, the only way you can access your MAAS deployed machines is via SSH public key.
NAT configuration For cloud-init similar setup, VMs on the maas libvirt network must be able to reach the wider network. Either of the following conditions should be met:
The bridge the VMs are attached to must include one of the host’s physical network interfaces on the appropriate network;
NAT must be enabled.
In this case the user uses NAT approach and sets the gateway to 10.7.7.13, which is the MAAS server’s IP. The command below is for your reference:
The upper settings will enable the IP forward of the host, then you can create NAT rules. The command line options are vary from firewall-cmd to iptables, this is an example of using firewall-cmd:
Alternatively, a simple line of iptables will do the job too:
Note
Zenlayer bare metal cloud doesn’t provide direct access to the instances' IPMI, therefore you cannot use IPMI interface call to manage instances.
All the Zenlayer bare metal instances have the PXE boot enabled on LAN interface, which is the default BIOS settings. After the OS installation, the LAN interfaces are repurposed as instances' “private interface“ for users.
Tip
You can bring up one instance with PXE installed and make it listening on the private VLAN, which will be your controller, then order the any other instances with “No OS“ option. Connecting the instances' private interfaces to your PXE connected VPC/VLAN, and triggering a reboot, the instances will start a PXE booting as the private interface is set as first boot option by default. This could be regarded as your own “bare metal level PXE OS installation“ .
You can deploy a new KVM server by checking Register as MAAS-managed KVM Host for a MAAS managed bare metal instance. See Add a KVM host for details.
While in this case, the OS has been installed with Zenlayer console, and this is a proof-of-concept type build, so a manual way needs to be explained.
Install the qemu-kvm with a command line:
This is the original ip a output:
lo/lan0/wan0 are for the bare metal servers loopback/LAN/WAN interface.
Configure the bridge settings
The brctl show output shows the virbr0 is the default bridge created by KVM, and the interface named virbr0-inc is bridged.
Assume the user wants to bridge the VM’s interface to the host (hypervisor)'s LAN interface, then the VM can get the IP address assigned by MAAS server’s DHCP (10.7.7.x/24 subnet). Change is also required in the KVM default network settings to make sure any new VM will use this bridge:
This will remove the original IP from lan0 to virbr0. Run the brctl to confirm the lan0 is added to virbr0:
The following ip a command is to verify the IP address settings:
Edit the libvirt default network profile The MAAS first checks for the existence of a libvirt network named maas. The maas network should have DHCP disabled in favor of MAAS-enabled DHCP to allow your VMs to network boot. If MAAS cannot find a maas network, it will fall back to libvirt’s default network. It is important to make the VM network able to communicate with MAAS, which could be done by either creating a maas libvirt network profile or modifying the KVM default network profile. Assume the user chooses to modify the default profile:
The original setting has KVM DHCP enabled for subnet 192.168.x.x, as the VM will be bridged with virbr0 to get IP assignment for MAAS DHCP, change it as below:
Add MAAS users key for SSH access The SSH access is used by MAAS server for management. To make it easy, grant the root access for maas server by adding the maas account’s public key into the .ssh/authorized_keys file which locates in the root home folder.
Tip
You can add the maas public key to the corresponding account for your libvert KVM management account, if you prefer to have tight restriction on different users' privilege.
After adding the maas user's public key to the root authentication of hypervisor, you can run a ssh test to make sure it works:
The bash command is the approach that the MAAS server communicates with KVM hypervisors. Make sure it works:
The 10.7.7.11 is the Test802Q-01 instance’s LAN IP address, and the output would be like:
This output is exact the same as running virsh list --all on hypervisor, which means the communication between MAAS and Hypervisor (KVM in this case) is working great.
Register the hypervisor Go to KVM tab and register the KVM hypervisor.
In this case, the virsh address is qemu+ssh://root@10.7.7.11/system. The MAAS will use the maas account to log in to hypervisors, instead it is mapping to the root account on the hypervisor by adding the public key into /root/.ssh/authorized_keys.
After the registration, you can choose Refresh from Take action drop-down menu to retrieve the virsh VM ID or the status of the hypervisor.
Compose VM In this case, both 10.7.7.11 and 12 servers are added into the MAAS, you can click on one of them and choose Compose from Take action drop-down menu to generate your first VM:
Deploy VM After the compose stage, the VM will be shut down and ready for deployment. Then you can deploy the server by choosing the OS and set other options (e.g. cloud-init).
You have set up the NAT rule on MAAS server and bridge the VM network into the 10.7.7.x while using DHCP to set the gateway. By checking the cloud-init option you can let the VM to grab init settings after the initial setup, even your cloud-init metadata server is sitting outside of Zenlayer networks.
During the commission and deployment, you can also enable VNC access to the VM, the screen output would be same as most of the PXE bootings:
This is the final view of all the VMs deployed:
DHCP IPs are assigned for each of VMs. If you execute the brctl show command to check the network settings, you will see all the new created vnet0~2 virtual network adapters are all bridged to virbr0:
Log in to VM and verify settings Now your VPC is up and running, the ens3 is the network interface name in VM, while the vnet0~2 are the interface names for hypervisor:
Note
MAAS enables SSH key login as the only option after an OS deployment of VM. Please upload your SSH key before deploying any server.
In this case, Ubuntu is chosen for deployment and the default user will be “ubuntu”, while 10.7.7.6 is the DHCP assigned IP address for VM 088, which is running over the hypervisor ”Test802Q-02”. Make an SSH login test with the private key pair for MAAS admin:
And the ip r shows the default route is set to MAAS server which is working as an NAT gateway:
Another best practice is to re-route the VM’s default gateway to your hypervisor, as Zenlayer bare metal instance has one Internet IP assigned by default, you can utilize the hypervisor’s network for the Internet access of all VMs.
Assume the VM 088 is running over the hypervisor “Test802Q-02”, you can go to Zenlayer console to check its Internet connectivity plan and status. It shows a valid IPv4 address with 10 Mbps flat rate data plan. As the hypervisor’s LAN interface “lan0” is bridged with VM’s interface via “virbr0”, you can create another NAT rule on the Test802Q-02 host to make it work as the gateway for all the VMs running on it, so that the traffic will be balanced between different Zenlayer bare metal instances instead of MAAS NAT gateway.
A major concern is that the Zenlayer bare metal instance is set to PXE boot by default, which cannot be changed due to the Zenlayer platform restriction. If you reload the bare metal instance, it will be picked up by MAAS PXE for registration or commission. As this is a proof-of-concept type testing and you use MAAS while its PXE service is enabled by default, another option is to disable the PXE of MAAS and enable it only when you are doing the VM provisioning, or you can configure a block list on your DHCP server to skip the DHCP request from bare metal instance, and the MAAS DHCP settings is for your reference:
Also a single line ebtables command with MAC address block list will work too:
You may also encounter some issues during the commissioning of VMs over the MAAS, especially when the VM interface is not assigned to correct bridge. The correct bridge should have bare metal instance’s lan0 included, usually the VM creation will call the default profile and create vnet0/2/3… in that bridge. The solution is to modify the VM profile and change the network interface part:
Another case is the default MAAS composed VM uses spice as the graphic option, somehow the VM console is unaccessible via virsh console [VM name], maybe the system lacking some lib. The solution is to switch to VNC.
Last issue is more like a reminding — remember to set VNC listening on 127.0.0.1 instead of 0.0.0.0. As every Zenlayer bare metal instance has one Internet IP by default, making VNC listening on Internet may end up your security being compromised.
