Can't start a VM till the lease expires

[gheibia]

I'm using Minikube with KVM through libvirt. Every time I stop the VM (which is the Minikube cluster's VM), I can't start it till the lease is expired.

Here is some detail:

Minikube version: v0.17.1
Environment:

• OS: Ubuntu 16.04.2 LTS (Xenial Xerus)
• VM Driver: KVM, libvirtd (libvirt) 1.3.1
• VM image: ~/.minikube/machines/minikube/boot2docker.iso

What happened:
Whether or not the Minikube cluster's VM is properly stopped, I can't start it until the DHCP lease is expired.

~$ minikube start --vm-driver="kvm" --insecure-registry=192.168.42.22:80 --logtostderr --v=7
Starting local Kubernetes cluster...
Starting VM...
I0411 17:29:13.318825   12960 cluster.go:67] Machine exists!
Found binary path at /usr/local/bin/docker-machine-driver-kvm
Launching plugin server for driver kvm
Plugin server listening at address 127.0.0.1:44304
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetState
(minikube) DBG | Getting current state...
(minikube) DBG | Fetching VM...
I0411 17:29:13.332112   12960 cluster.go:74] Machine state:  Stopped
(minikube) Calling .Start
(minikube) DBG | Starting VM minikube
(minikube) DBG | GetIP called for minikube
(minikube) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(minikube) DBG | IP address: 192.168.42.11
(minikube) DBG | Unable to locate IP address for MAC 52:54:00:37:5b:48
(minikube) Calling .GetConfigRaw
Waiting for SSH to be available...
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG | GetIP called for minikube
(minikube) DBG | Failed to retrieve dnsmasq leases from /var/lib/libvirt/dnsmasq/docker-machines.leases
(minikube) DBG | IP address: 192.168.42.11
(minikube) DBG | Unable to locate IP address for MAC 52:54:00:37:5b:48
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) DBG | AK: resolvestorepath: /home/amir/.minikube
(minikube) DBG | AK: resolvestorepath: /home/amir/.minikube
(minikube) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /home/amir/.minikube/machines/minikube/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none [email protected] -o IdentitiesOnly=yes -i /home/amir/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
exit 0
SSH cmd err, output: exit status 255: 
Error getting ssh command 'exit 0' : Something went wrong running an SSH command!
command : exit 0
err     : exit status 255
output  :

It seems that this driver can't find the right lease file. That's the only error I see.

What you expected to happen:

Find the correct lease file and start without an error.

How to reproduce it:

• Start minikube with KVM.
• Stop it.
• Try to start it before lease is expired. I don't know what the default expiry time is. But try to start it right after.

Additional info:

https://bugzilla.redhat.com/show_bug.cgi?id=1228631

It seems that libvirt is changed to read lease records from custom made files (named after interface). But I think you already knew that.

[aaron-prindle]

What version of the kvm driver are you using?

[gheibia]

~$ kvm --version
QEMU emulator version 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.10)

[r2d4]

@gheibia I think he was wondering what version of docker-machine-kvm did you download? This should have been fixed in v0.8.2 of this driver

[gheibia]

@r2d4 I don't think I've downloaded this directly. My guess is that this would be part of the docker-machine. If so, here is what I got:

$ docker-machine --version
docker-machine version 0.10.0, build 76ed2a6

[gheibia]

After a bit of digging, it seems that I'm using 0.7.0. I couldn't determine the version through the command line (Plugin binaries are not intended to be invoked directly), but I generated the MD5 checksum for the plugin's binary (519d29300df740a73c90d048e5df43fb) which matches 0.7.0. I'll update the plugin and try again.

[gheibia]

I can confirm that this has indeed been resolved in 0.8.2. I manually replaced my older plugin with the new one and dropped and recreated the cluster. I can now stop and start the VM right after. I still see the error message about not finding the old lease files, but at least the cluster starts up.

Thanks so much for the help.

[gheibia]

Having this fixed though I now observe that the VM's IP changes between restarts. I wonder if there is a workaround for this. Do I need to open another ticket for it?

[gheibia]

After updating minikube to v0.18.0 the problem returned. I figured if I update the docker-machine-kvm to its latest version (https://github.com/dhiltgen/docker-machine-kvm/releases/tag/v0.10.0), the problem might disappear, but it didn't. Basically, the VM won't start till the lease expires.

[gheibia]

There is a workaround for this:

1. Stop minikube
2. Stop libvirtd service (sudo service libvirtd stop)
3. Kill all dnsmasq processes (sudo pkill -f dnsmasq)
4. Edit dnsmasq's files and remove all traces of minikube (/var/lib/libvirt/dnsmasq/virbrxxx.status)
5. Restart the host to start everything clean.

[gbraad]

not reproducible with current versions of the driver plugin

[cchet]

I experience the same issue since a few days.