MicroVMs
Understanding common MicroVM boot/run errors
Data and logs for each MicroVM will be stored under /var/lib/flintlock/vm
, with
the full path for a MicroVM being /var/lib/flintlock/vm/NAMESPACE/NAME/UUID
.
Under that directory are the following files:
firecracker.cfg
: the configuration set byflintlock
forfirecracker
firecracker.log
: logs from the firecracker processfirecracker.metrics
: metrics from the firecracker processfirecracker.pid
: the pid of the running firecracker processfirecracker.stderr
: MicroVM boot errorsfirecracker.stdout
: MicroVM boot logsmetadata.json
: MicroVM boot userdata
MicroVM network configuration failure/timeout
When MicroVMs boot, they will be assigned an address from wherever you have a dhcp server. This may be your home router or a dedicated pool you have configured for a VLAN.
If you see the following logs, then it means this has failed.
A start job is running for Wait for Network to be Configured (38s / no limit)
... etc for a couple of mins. but really if it is going to work it will do
... so within 10 seconds
Failed Wait for Network to be Configured.
Steps:
- Check that the host can ping the router. If not, and you have access to the router, disconnect the device (turn it off) and remove the previous record from the router's list. Reconnect the device and try pinging again.
- Ensure that the correct interface has been configured in
/var/lib/flintlockd/config.yaml
.parent-iface
should be set to a wired interface (egeth0
). If you are using flintlock in bridge mode,bridge-name
should be a bridge which is mastered to a parent interface with internet access. - On the MicroVM's host device run
tcpdump -i <parent interface name> -vv
while creating a MicroVM. Look for dhcpDiscover
,Offer
,Request
andACK
messages. If you see none, it means the MicroVM cannot reach your dhcp server at all. If you see the first 2 but not the last, it means the dhcp server has offered an IP by the client for some reason does not receive or accept that. From here you are debugging based on your particular device and networking setup.
Cannot SSH into MicroVM
- Check that you set a public SSH key in the spec
user-data
or the cluster manifest. - Check the
firecracker.stdout
boot logs of the MicroVM to make sure you have the correct IP, or that one has been set at all (see the above section).
MicroVM has not started
If you create a MicrmVM but cannot see a firecracker
process or the boot logs
at /var/lib/flintlock/vm/NS/NAME/UID/firecracker.stdout
are empty, try these steps:
- Look at
journalctl -fu flintlockd.service
and for the line:If this is not present, find what the last action for that UID was."finished executing plan" controller=microvm execution_id=UID execution_time=4m3.498577899s num_steps=6 plan_name=microvm_create_update
noteOn the first create with a new kernel or OS image, it can take a while for containerd to pull it down.
- Check the MicroVM error logs at
/var/lib/flintlock/vm/NS/NAME/UID/firecracker.stderr
- Check the
firecracker
logs at/var/lib/flintlock/vm/NS/NAME/UID/firecracker.log
If you are using ARM images, ensure that you have updated the kernel.filename
in the
manifest to be boot/image
. For x86
this should be boot/vmlinux
.