I have been closely monitoring the expanded output when these image retrievals fail, and it appears to be an on-going issue after all. It appears that the images DO exist in the storage facility but are failing to download at random. No error is provided by the storage provider. This is occurring at a rate of about 1 in 150,000 requests. For some reason, it seems to be happening more often with a small subset of users. That may lend a clue if we are able to pinpoint some similarities between the requests and/or images. However, it may also be coincidence, based on usage and a specific type of integration.

We have opened a ticket with our storage provider and will work with them to determine the root cause of the failures and correct the issue.

There may be more than one issue going on here, but I have identified one major cause that may be the only culprit. Several months ago, I made a slight change to the "Rolling Refresh" query that controls which "stale" requests to refresh (in order to keep our cached images fresh).

This change included refreshing certain error types, in case broken sites may be working after some time, but I did not account for those error types in the update query. So the system was treating them as if they were "good" and the XML response was telling users to download images that should not exist --hence the error. I have corrected this now and it should not happen again.

If we continue to see errors for some other reason, then I will have to dig deeper to see why. However, I have a feeling that this may be the fix that was needed.

I have been monitoring this issue for weeks, and it appears that there are still reasons that cause these errors. After in-depth testing and enabling logging in our third-party storage provider, it appears to be a systematic issue with the third-party storage provider (possibly related to our high rate of access).

There are two conditions that have been causing the errors we log as "Image Retrieval Error":

1. Cached Image Missing - for whatever reason, the image no longer exists in the third-party cache. This should not be possible, so we have been collecting logs and will open a ticket with the storage provider. In the meantime, we have modified the code to detect this specific condition and log it in our system as "Cached Image Missing" from now on (instead of "Image Retrieval Error"). This condition will result in an automatic, free refresh request for that request.

2. Connection Error / No Response - for an unidentified reason, the remaining errors are happening with no data response from the storage provider to indicate what went wrong. Subsequent attempts succeed, so the image exists in cache. These failures are intermittent, so we have been collecting logs for awhile and will open a ticket with our storage provider to attempt to identify the root cause of this issue and prevent it. These errors will continue to be logged in our system as "Image Retrieval Error".

I suspect that our high rate of access may be to blame, but our storage provider says that they can handle quite a bit more volume than we currently generate. We have also worked with them to optimize the image per storage volume distribution, but that has had no noticeable effect.

We will continue to work on this issue to find a suitable resolution.