The new capture process was very successful and resolved a lot of lingering issues and new issues that had shown up for no discernible reason. The issues we noted were beyond our control, so we felt it necessary to start over.

For the past month, we have been running on about 15% of our normal capacity, using only the new capture generator technology. It was enough to keep up with demand, so we used the opportunity to provide more load to the new generators for testing and monitoring.

As of today, all capture generators have updated and launched.

The only error we found with the new capture process that may have been present in the old process (but went undetected) is that a number of URLs were failing with a blank image captured. This was happening on very slow loading sites that, for some unknown reason, take even longer from our primary hosting datacenters. It may be something specific to the network of our primary hosting company but they refuse to admit there is a problem because the delay in loading the sites only occurs in our capture process (ping, curl, traceroute, etc all load the sites very quickly). However, the sites load quickly in our secondary hosting datacenters, so that tells us that it is not our process. Perhaps there is some conflict with the primary datacenter related to our process, but I have no clue where to even start looking for that possible culprit.

In any case, we have no control over this issue and resorted to extending the timeout to allow more time for those types of sites to load. This ensures a proper capture instead of the BLANK_DETECTED error. We identified 250,000 BLANK_DETECTED captures in our cache, so I have set them all to retry --in case a good many are valid but suffered from this issue.

Aside from that, our capture success rate should be at an all-time high and throughput per generator appears to be increased as well. Under heavy load, the capture generators perform "as fast" as the old process. Under light load, they perform 200-300% faster.