There seems to be an issue with the update from Joomla 2.5.4 to 2.5.5 or 2.5.6 that causes sendmail to stop functioning in less than PHP v5.3.10. This article explains how to recover from the "Could not instantiate mail function." error after an update.
Joomla 2.5 is no longer supported.
I've not fully figured out what is wrong with the update. I know it's realated to your version of PHP and the phpmailer.php file. I assume it's a PHP version issue.
To fix this error you can revert the file libraries/phpmailer/phpmailer.php back to the 2.5 version (2.5.1 is the version that doesn't work). To restore other mail files (I've heard SMTP is broken too) just replace the whole libraries/phpmailer/ directory with the version from 2.5.4. I looked up the version in git as it's easier than grabbing the zip.
That's it you should have mail back and running on PHP less than v5.3.10.
Also this error can come up if you have safe mode on. Make sure PHP safemode is off.