Last week, we started to experience a really bizarre class exception that started to appear every so often. It was coming from within the Grizzly (1.8) framework when it was attempting to open the java.log.Logger to report an exception.
The full stack trace seemed to suggest there was a corruption in the JAR file which was even more perplexing. The JDK had not been changed or updated, and the deployed JAR files had not actually changed in a number of days. So why now?
After some looking around on t'internet, I could not find anyone with the same stack trace, so that wasn't inspiring me with confidence.
A quick post on the Grizzly mailing list, didn't yield any assistance, although some great suggestions came back. Jeanfrancois kindly built me a special 1.8 JAR file that had the logging removed, instead pushing the error to the System.err.
This proved to useful. Another error started to popup, "too many open files", which was indictative of a server under extreme load, or that, that something wasn't closing down file handles. Although don't get sidetracked that files mean actual files, under Linux, this includes network sockets.
I have since increased the number of allowed open file handles and we haven't seen the error pop its head up again, but monitoring it closely.
let me thank Jeanfrancois for his fast turnaround and assisting in work through this; he's a classic example of why open source works.