Last week we upgraded Cisco Prime Infrastructure to its latest security patch level (3.7 update 3). So the install kicked off and the reboot happened but there was no login on the console after linux booted. The console just showed a flashing cursor. You could SSH to the VM and access the shell, so networking services and the general OS was booted and active. Trying the usual ncs start would just hang the VM for ages. OK, something was not right here.

Heres what I did the diagnose and fix the issue:
1. Login to the VM with SSH and drop into the shell (with the shell command).
2. Run the command df (Linux Disk Free utility). It showed that /var was 100% full.
3. Check for following files in /var/log (this will list the size they take):
# ls -lhrt /var/log/wtmp
# ls -lhrt /var/log/secure
4. Check if more secure files are created:
#cd /var/log #ls -lrt *secure*
5. Remove the files with sudo rm /var/log/wtmp and sudo rm/var/log/secure. It is also ok to remove any file named secureX where X is a number (secure5 for example).
6. Reboot the VM appliance.
In my case the issue was /var/log/wtmp which was over 3GB in size.

I learnt two things from this:
1. That the ncs application (Cisco Prime Infrastructure itself) will not start if /var is full.
2. With the VM appliance if there is a disk issue you’ll see a flashing cursor on console as well.