My Use Case: Troubleshooting “No valid host was found”
Error: Failed to perform requested operation on instance “Waghoba”,
the instance has an error status: Please try again later [Error: No
valid host was found. There are not enough hosts available.].
Troubleshooting Openstack is not a easy Job. Logs, logs and more logs , all you need to look at those voluminous log at different locations.
Typically when encountered with “No Valid Host Found ” error you need to check the filters. In this example the error was caused due to Insufficient Disk ” space or inappropriate disk partition. Lets see how I investigated and fixed the problem.
Nova uses filters RetryFilter, AvailabilityZoneFilter, RamFilter,DiskFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, CoreFilter ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter. These filters are defined in /etc/nova/nova.conf which actually does the scheduling decisions. Its not necessarily that we use all the filters some of which can be disabled.
Look into the /var/logs/nova folder. I am focusing on nova-conductor log.
Try to search the words “returned 0 hosts” in the log location /var/log/nova
[root@netcon etc(keystone_admin)]# cd /var/log/nova/
[root@netcon nova]# ls
nova-api.log nova-cert.log nova-conductor.log nova-consoleauth.log nova-manage.log nova-novncproxy.log nova-scheduler.log
[root@netcon nova(keystone_admin)]# cat *.log | grep “returned 0 hosts”
2017-11-24 21:53:47.863 1245 INFO nova.filters [req-394b09de-c52d-460d-925c-ba9c1220f09b d9011c1e30864e489e12305b7aef6cd3 40bd32cbd522405a9b236fc738c003dd – – -] Filter ComputeFilter returned 0 hosts
2017-12-04 11:50:52.780 1222 INFO nova.filters [req-07a1779a-eea6-454a-97b8-bc34ae03d501 d9011c1e30864e489e12305b7aef6cd3 40bd32cbd522405a9b236fc738c003dd – – -] Filter DiskFilter returned 0 hosts
2017-12-04 14:00:25.336 18105 INFO nova.filters [req-417277d5-e61f-4096-8e24-8a4196c6718f d9011c1e30864e489e12305b7aef6cd3 40bd32cbd522405a9b236fc738c003dd – – -] Filter DiskFilter returned 0 hosts
Clearly the error is pointing towards the “Disk space/ Partition”. Filter DiskFilter is the pointer.
[root@compute ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 50G 1.7G 40G
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 8.5M 16G 1% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/mapper/centos-home 399G 33M 399G 1% /home
/dev/sda1 497M 186M 312M 38% /boot
tmpfs 3.2G 0 3.2G 0% /run/user/0
What does the Dashboard says ?
I was provisioning a Brocade-Vyatta5600 Router VM (VM name Waghoba) with the default large flavor. The compute nodes installation mode was made t with auto disk partition(/dev/mapper/centos-root 40GB).
The Dashboard was displaying Hypervisor summary showing 40GB as local storage which it was picking from the partition of the compute node. This value was greater than the flavor applied for the VM creation as a result the “Diskfilter” was not allowing to create the VM. The logs and the key word “”returned 0 hosts” grepped from it clearly indicated the problem which was fixed latter by changing the partition on the compute node.