About the author
Martin Buchleitner is a Senior IT-Consultant for Infralovers and for Commandemy. Twitter github LinkedIn
See all articles by this authorAutomated VMWare Templates with HashiCorp Packer
Within all the cloud providers you get automated built template based virtual machines - what about doing this on your own to build your custom Templates for VMWare vSphere? In this case HashiCorp Packer is the tool you want to go for. This posting will go through the main concepts of creating a packer based virtual machine template
What is Packer?
Packer is an open source tool for creating identical machine images for multiple platforms from a single source configuration. Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer does not replace configuration management like Chef or Puppet. In fact, when building images, Packer is able to use tools like Chef or Puppet to install software onto the image.
A machine image is a single static unit that contains a pre-configured operating system and installed software which is used to quickly create new running machines. Machine image formats change for each platform. Some examples include AMIs for EC2, VMDK/VMX files for VMware, OVF exports for VirtualBox, etc.
Advantages of Packer
Packer images allow you to launch completely provisioned and configured machines in seconds, rather than several minutes or hours. This benefits not only production, but development as well, since development virtual machines can also be launched in seconds, without waiting for a typically much longer provisioning time.
Because Packer creates identical images for multiple platforms, you can run production in AWS, staging/QA in a private cloud like OpenStack, and development in desktop virtualization solutions such as VMware or VirtualBox. Each environment is running an identical machine image, giving ultimate portability.
Packer installs and configures all the software for a machine at the time the image is built. If there are bugs in these scripts, they’ll be caught early, rather than several minutes after a machine is launched.
After a machine image is built, that machine image can be quickly launched and smoke tested to verify that things appear to be working. If they are, you can be confident that any other machines launched from that image will function properly.
Packer makes it extremely easy to take advantage of all these benefits.
Ubuntu based example
Install Packer
To start with Packer download the prebuilt binary. After installing Packer, verify the installation worked by opening a new command prompt or console, and checking that packer is available:
$ packer
usage: packer [--version] [--help] <command> [<args>]
Available commands are:
build build image(s) from template
fix fixes templates from old versions of packer
inspect see components of a template
validate check that a template is valid
version Prints the Packer version
ESXi Configuration
To use the upper template with VMWare ESXi 6 and later you must perform some modifications on the ESXi host. The ESXi host needs a little bit of configuration to allow Packer to work. Packer communicates over SSH, so first we need to open that. Secondly, we’ll enable an option to discover Guest IPs from the Host and then finally allow VNC connections remotely.
Enable SSH
Inside the web UI, navigate to “Manage”, then the “Services” tab. Find the entry called: “TSM-SSH”, and enable it. You may wish to enable it to start up with the host by default. You can do this inside the “Actions” dropdown (it’s nested inside “Policy”).
Enable “Guest IP Hack”
Run the following command on the ESXi host:
esxcli system settings advanced set -o /Net/GuestIPHack -i 1
This allows Packer to infer the guest IP from ESXi, without the VM needing to report it itself.
Open VNC Ports on the Firewall
Packer connects to the VM using VNC, so we’ll open a range of ports to allow it to connect to it.
First, ensure we can edit the firewall configuration:
chmod 644 /etc/vmware/firewall/service.xml
chmod +t /etc/vmware/firewall/service.xml
Then append the range we want to open to the end of the file:
<service id="1000">
<id>packer-vnc</id>
<rule id="0000">
<direction>inbound</direction>
<protocol>tcp</protocol>
<porttype>dst</porttype>
<port>
<begin>5900</begin>
<end>6000</end>
</port>
</rule>
<enabled>true</enabled>
<required>true</required>
</service>
Finally, restore the permissions and reload the firewall:
chmod 444 /etc/vmware/firewall/service.xml
esxcli network firewall refresh
Template Definition
The configuration file used to define what image we want built and how is called a template in Packer terminology. The format of a template is simple JSON. JSON struck the best balance between human-editable and machine-editable, allowing both hand-made templates as well as machine generated templates to easily be made.
We’ll start by creating the entire template, then we’ll go over each section briefly. Create a file example.json and fill it with the following contents:
{
"builders": [{
"name": "Ubuntu-19.04",
"type": "vmware-iso",
"vm_name": "Ubuntu-19.04",
"guest_os_type": "ubuntu-64",
"tools_upload_flavor": "linux",
"headless": true,
"iso_urls": ["iso/ubuntu-19.04-server-amd64.iso",
"http://cdimage.ubuntu.com/ubuntu/releases/19.04/release/ubuntu-19.04-server-amd64.iso"
],
"iso_checksum": "7e8a0d07522f591dfee9bc9fcd7c05466763161e6cb0117906655bce1750b2fa",
"iso_checksum_type": "sha256",
"cpus": 2,
"memory": 4096,
"disk_size": 20140,
"boot_wait": "10s",
"boot_command": [
"<esc><wait>",
"<esc><wait>",
"<enter><wait>",
"/install/vmlinuz",
" initrd=/install/initrd.gz ",
"auto=true ",
"url=https://raw.githubusercontent.com/infralovers/packer-preseed/master/ubuntu.cfg ",
"fb=false ",
"auto=true ",
"language=en ",
"locale=en_US ",
"priority=critical ",
"keymap=us ",
"netcfg/get_hostname={{ .Name }} ",
"netcfg/get_domain=vm ",
"debconf/frontend=noninteractive ",
"debian-installer/country=AT ",
"console-setup/ask_detect=false ",
"console-keymaps-at/keymap=us ",
"DEBCONF_DEBUG=5 ",
"<enter>"
],
"ssh_username": "vagrant",
"ssh_password": "vagrant",
"ssh_port": 22,
"ssh_wait_timeout": "20m",
"shutdown_command": "echo 'shutdown -P now' > shutdown.sh; echo 'vargant'|sudo -S sh 'shutdown.sh'",
"vnc_disable_password": true,
"format": "vmx",
"remote_type": "esx5",
"remote_host": "{{user `esxi_host`}}",
"remote_datastore": "{{user `esxi_datastore`}}",
"remote_username": "{{user `esxi_username`}}",
"remote_password": "{{user `esxi_password`}}",
"keep_registered": true,
"skip_export": true
}]
}
You can also start building images with VMWare Fusion when you remove the remote_* keys in the json. When doing the initial work this can be quite useful to keep the loop tight, because debugging debconf or kickstart files is easier going the local devloop In the Ubuntu based example the bootswitch DEBCONF_DEBUG is used to output on the installation log when a settings is found that is not set or set to an invalid value.
Building the image
With a properly validated template, it is time to build your first image. This is done by calling packer build with the template file. The output should look similar to below. Note that this process typically takes a few minutes. Because the template uses variables for the esxi specific data, these must be set on build time with the -var argument.
packer build -var "esxi_host=YOUR-ESXI" -var "esxi_datastore=YOUR_DEFAULT_DATASTORE" -var "esxi_username=ESXI_USERNAME" -var "esxi_password=ESXI_PASSWORD" ubuntu.json
Provisioning Packer Templates
The upper command would just generate a default installed Ubuntu 19.04 image using a preseed file which just generates a user who is able to use sudo.
So the next step in this setup is to do some custom provisioning. For this packer is able to launch provisioners. Using these methods you are able to:
- Setup the provisioning ( e.g. install ansible and dependencies )
- Do custom provisioning ( e.g. with ansible playbook )
- Remove provisioning tools ( e.g. remove ansible )
The full template will like this:
{
"builders": [{
...
}],
"provisioners": [{
"type": "shell",
"execute_command": "echo 'vagrant' | {{.Vars}} sudo -S -E bash '{{.Path}}'",
"script": "scripts/setup.sh"
},
{
"type": "ansible-local",
"playbook_dir": "ansible",
"playbook_file": "ansible/main.yml",
"extra_arguments": ["--extra-vars \"root_password={{user `root_password`}} user_password={{user `user_password`}}\""]
},
{
"type": "shell",
"execute_command": "echo 'vagrant' | {{.Vars}} sudo -S -E bash '{{.Path}}'",
"script": "scripts/cleanup.sh"
}
]
}
The layout of the local is like
|-- ansible
| |-- ansible.cfg
| |-- main.yml
| |-- roles
|-- scripts
|-- ubuntu.json
`-- variables.json
Within ansible/main.yml you can define your playbook to install default tools and libraries ( e.g. cloud-init ) and set custom configuration items.
Packer Post Processing
When running the upper build process the result is a customized VMWare machine, which is ready to run - but we wanted to get a VMWare template to produce virtual machines with this build. For this Packer has Post Processors in place. In our use case we gonna use the vsphere post processor and also the vsphere-template processor which finally generates our template. The vsphere post processor halts the previous generated virtual machine and tranfers it to the specified directory. Afterwards the vsphere template post processesor transforms the virtual machine into a machine template for later usage.
{
"builders": [{
...
}],
"provisioners": [{
}],
"post-processors": [
[{
"type": "vsphere",
"cluster": "{{user `esxi_vsphere_cluster`}}",
"host": "{{user `esxi_vsphere_host`}}",
"datacenter": "{{user `esxi_vsphere_dc`}}",
"username": "{{user `esxi_vsphere_username`}}",
"password": "{{user `esxi_password`}}",
"datastore": "{{user `esxi_datastore`}}",
"vm_name": "T_Ubuntu",
"vm_network": "VM Network",
"vm_folder": "/Templates",
"disk_mode": "thin",
"insecure": "true",
"overwrite": "true"
},
{
"type": "vsphere-template",
"host": "{{user `esxi_vsphere_host`}}",
"insecure": "true",
"datacenter": "{{user `esxi_vsphere_dc`}}",
"username": "{{user `esxi_vsphere_username`}}",
"password": "{{user `esxi_password`}}",
"folder": "/Templates"
}
]
]
}
