Troubleshooting_Openstack_Part1

My Use Case: Troubleshooting “No valid host was found”

Error: Failed to perform requested operation on instance “Waghoba”,

 the instance has an error status: Please try again later [Error: No 

valid host was found. There are not enough hosts available.].

Problem

 

Troubleshooting Openstack is not a easy Job. Logs, logs and more logs , all you need to look at those voluminous log at different locations.

Typically when encountered with “No Valid Host Found ” error you need to check the filters. In this example the error was caused due to Insufficient Disk ” space or inappropriate disk partition. Lets see how I investigated and fixed the problem.

Nova uses filters RetryFilter, AvailabilityZoneFilter, RamFilter,DiskFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, CoreFilter ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter. These filters are defined in /etc/nova/nova.conf which actually does the scheduling decisions. Its not necessarily that we use all the filters some of which can be disabled.

Look into the /var/logs/nova folder. I am focusing on nova-conductor log.

Try to search the words “returned 0 hosts” in the log location /var/log/novaLogs_See

[root@netcon etc(keystone_admin)]# cd /var/log/nova/

[root@netcon nova]# ls
nova-api.log  nova-cert.log  nova-conductor.log  nova-consoleauth.log  nova-manage.log  nova-novncproxy.log  nova-scheduler.log
[root@netcon nova(keystone_admin)]#  cat *.log | grep “returned 0 hosts”

2017-11-24 21:53:47.863 1245 INFO nova.filters [req-394b09de-c52d-460d-925c-ba9c1220f09b d9011c1e30864e489e12305b7aef6cd3 40bd32cbd522405a9b236fc738c003dd – – -] Filter ComputeFilter returned 0 hosts
2017-12-04 11:50:52.780 1222 INFO nova.filters [req-07a1779a-eea6-454a-97b8-bc34ae03d501 d9011c1e30864e489e12305b7aef6cd3 40bd32cbd522405a9b236fc738c003dd – – -] Filter DiskFilter returned 0 hosts
2017-12-04 14:00:25.336 18105 INFO nova.filters [req-417277d5-e61f-4096-8e24-8a4196c6718f d9011c1e30864e489e12305b7aef6cd3 40bd32cbd522405a9b236fc738c003dd – – -] Filter DiskFilter returned 0 hosts

Clearly the error is pointing towards the “Disk space/ Partition”. Filter DiskFilter is the pointer.

[root@compute ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  1.7G   40G  
devtmpfs                  16G     0   16G   0% /dev
tmpfs                     16G     0   16G   0% /dev/shm
tmpfs                     16G  8.5M   16G   1% /run
tmpfs                     16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/centos-home  399G   33M  399G   1% /home
/dev/sda1                497M  186M  312M  38% /boot
tmpfs                    3.2G     0  3.2G   0% /run/user/0

What does the Dashboard says ?

Hypervisor2

ts

 

Summary:

I was provisioning a Brocade-Vyatta5600 Router VM (VM name Waghoba)  with the default large flavor. The compute nodes installation mode was made t with auto disk partition(/dev/mapper/centos-root 40GB).

The Dashboard was displaying Hypervisor summary showing 40GB as local storage which it was picking from the partition of the compute node. This value was greater than the flavor applied for the VM creation as a result the “Diskfilter” was not allowing to create the VM. The logs and the key word “”returned 0 hosts” grepped from it clearly indicated the problem which was fixed latter by changing the partition on the compute node.

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *