PLEASE NOTE: This article may be out-of-date. Please visit the Support site for the latest information.

Capture Generator Migration

6 posts / 0 new
Letzter Beitrag
Bild von puravida
puravida
Jedi Warrior
Offline
Beigetreten: 09/01/2007
Besuchen puravida's Webseite

GOAL: To deploy a fully scalable platform for growth of our service.

After more than a year of work and many delays, we are finally ready to begin preparation for migration to the next iteration of our capture technology. Therefore, we are instituting a "code freeze" on the stable capture code (stable for years), as of now. No troubleshooting or code changes will be made, unless some unforeseen and widespread issue occurs.

The phases of this huge project are:

  1. Deploy a process to handle authentication and provisioning of generators automatically*
  2. Modify the storage method to take advantage of infinitely scalable storage solution
  3. Migrate the primary servers to a virtual SSD-based cluster that is infinitely scalable, distributed, and able to be geographically dispersed

*This will pave the way for nearly infinite scalability of our service and full automation will mean we have more time to devote to enhancements and improvements in the future.

The first phase should finally be completed and tested within the next week or so. After that, step 2 will most likely be completed by the end of Summer 2012. Step 3 will most likely be completed late in Q3 or Q4 of 2012.

IMPACT
Phase 1: We do not expect any downtime but there may be occasional, short periods of "offline for maintenance" during the transition and testing. This is especially true if we run into issues with capturing certain sites with the newer technologies, once we go live. However, the worst case impact would just be bad captures that we would be actively monitoring, troubleshooting, testing, and then refreshing for free.

Also, since we are hitting some technological limitations, we will be clearing our entire cache of captures and will capture every request, as it comes in, for a while. This may cause slight delays of minutes or hours, for the first few days after go-live.

Phase 2: We do not expect any downtime but there may be occasional, short periods of "offline for maintenance" during the transition and testing. If there are issues, we will roll back to the current storage solution while we troubleshoot and prepare to retry the deployment. In a worst case, we may end up clearing out the entire cache of captures repeatedly, until we get a bulletproof deployment and then let the system run full speed.

Phase 3: We do not expect any downtime, but there may be occasional, short periods of "offline for maintenance" during the transition..

GEEK-SPEAK
The long-term goal of all of this work is to allow us to greatly extend the retention policy at some point. This will be the first step towards growing our pre-captured screenshots of web pages to include every web page on the Internet. It is an ambitious, and admittedly impossible, goal because of the vast number of web pages and extraordinary growth of the Internet.

However, we have to start somewhere and achieving even 10% captures of the most trafficked web pages would mean less than 1% of requests would be considered new. As we approach that point, the need for capture generators will transition from new captures to just keeping requests refreshed.

The beauty of that lies in being easily quantifiable, which means that we can eliminate "New Requests" from our pricing model and confidently build the cost of generators into the service, overall. That will greatly simplify the sales process and will help to lower costs in the long run.

While these phases require a substantial initial investment and considerable increases to our operating expenses, we feel that this is a necessary evolution for our service. These added costs will be leveraged quickly, as our service continues to grow exponentially. So, we are betting on that continued growth. Eventually, the added costs will be a nominal portion of the overall operation and, with our loyal user's continued support, we believe that we will reach that goal and earn the freedom to give back to our valued customers!

Bild von puravida
puravida
Jedi Warrior
Offline
Beigetreten: 09/01/2007
Besuchen puravida's Webseite

To answer a couple of comments that came in: this message applies "generally" to our users.

There will be a smaller group who require more specific, customized solutions or high-usage of "custom sizes", etc. Those will most likely remain at the same prices and deliver on the same SLA as they do now. We cannot hope to capture every web page AND have it in every size. Capturing at an extremely large size and then scaling on-the-fly would work for "custom sizes" but not for full-length, so we just have to evaluate as we get to that point. It is a long way (years) away.

Bild von puravida
puravida
Jedi Warrior
Offline
Beigetreten: 09/01/2007
Besuchen puravida's Webseite

After 2 years of research, development, and testing; we are finally ready to move ahead.

Phase 1 is nearly complete and we have a few capture generators currently in a LIVE BETA. They are producing even better images, quicker and more reliably than our current production capture generators.

The new generators, surprisingly, resolve every single issue that we have had in our support queue. We had a 100% success rate so far. As we process a larger sampling, we will ensure that there are no issues. There should be no negative impact to launching the new BETA generators, but we will be closely monitoring for quality control and addressing issues as quickly as possible.

If all goes well, we will be transitioning all of our capture generators to the new technology. That should all take place by the end of this month.

Bild von puravida
puravida
Jedi Warrior
Offline
Beigetreten: 09/01/2007
Besuchen puravida's Webseite

Turns out that a change, that took place right before our major LIVE BETA, broke Flash capturing. So requests that were new or refreshed over the past 48 hours will probably have missing Flash. There is no way for us to know that a site has Flash, so if you come across one; just refresh it.

We will double check Flash before we launch the next LIVE BETA.

We are very close and result from the last LIVE BETA, aside from Flash issue, were very encouraging. These new generators will be an order of magnitude better than what we have now.

Bild von puravida
puravida
Jedi Warrior
Offline
Beigetreten: 09/01/2007
Besuchen puravida's Webseite

UPDATE: The new generators performed so much better than we could have hoped they would. We just finished the next step in testing, which was a full LIVE run that included decommissioning all "old" production generators, handling all incoming requests (roughly 275,000 in the last 24 hours), all while processing through 300,000 broken requests.

The broken requests are the toughest to process because they hang the render engine processes and cause numerous issues. With 300,000 broken requests, we have seen it take 10 days to clear the queue. This time, though, the new generators cleared the queue within the 24 hour period of testing. We expected it to take at least a few days, so this is really encouraging.

So, as of tonight, we are officially LIVE on the new generator technology and have integrated a service that fully automates and manages the entire process of provisioning and decommissioning the number of generators based on queue and demand. This means that PHASE I has made our service infinitely scalable from the standpoint of capturing screenshots.

The next two phases will allow us to grow our storage to extremes and will allow our primary servers to scale infinitely as well. At that point, we will be in a position to handle the entire world's screenshot needs, without having anymore foreseen growing pains.

The technical implementation was very difficult and this roll-out is the culmination of over two years of planning, testing, and development. However, the other huge hurdle was money. The services required to fully automate the billing and the provisioning to infinite scalability are not cheap by any means. So it says something that our users have helped us to fund the $150,000 in research & development and more than $4,500/month in extra automation costs, so far, to make this a reality. We are very pleased to be able to invest in the service in order to get to this point.

By next year, our focus will turn to sales, optimization, integrations, and lowering costs for our loyal users. We appreciate everyone's continued support. Thank you! Smile

Bild von puravida
puravida
Jedi Warrior
Offline
Beigetreten: 09/01/2007
Besuchen puravida's Webseite

After the latest upgrade to all of our generators, it appears that they are processing at twice the efficiency over last week. This is great news, as it will mean shorter capture times and faster turn-around on refreshes.

Topic locked

ShrinkTheWeb® (About STW) is another innovation by Neosys Consulting

Kontakt | PagePix Benefits | Learn More | Our Partners | Datenschutzrichtlinie | Nutzungsbedingungen

©2014 ShrinkTheWeb. Alle Rechte vorbehalten. ShrinkTheWeb ist ein eingetragenes Handelszeichen von ShrinkTheWeb.