GOAL: To deploy a fully scalable platform for growth of our service.
After more than a year of work and many delays, we are finally ready to begin preparation for migration to the next iteration of our capture technology. Therefore, we are instituting a "code freeze" on the stable capture code (stable for years), as of now. No troubleshooting or code changes will be made, unless some unforeseen and widespread issue occurs.
The phases of this huge project are:
- Deploy a process to handle authentication and provisioning of generators automatically*
- Modify the storage method to take advantage of infinitely scalable storage solution
- Migrate the primary servers to a virtual SSD-based cluster that is infinitely scalable, distributed, and able to be geographically dispersed
*This will pave the way for nearly infinite scalability of our service and full automation will mean we have more time to devote to enhancements and improvements in the future.
The first phase should finally be completed and tested within the next week or so. After that, step 2 will most likely be completed by the end of Summer 2012. Step 3 will most likely be completed late in Q3 or Q4 of 2012.
IMPACT
Phase 1: We do not expect any downtime but there may be occasional, short periods of "offline for maintenance" during the transition and testing. This is especially true if we run into issues with capturing certain sites with the newer technologies, once we go live. However, the worst case impact would just be bad captures that we would be actively monitoring, troubleshooting, testing, and then refreshing for free.
Also, since we are hitting some technological limitations, we will be clearing our entire cache of captures and will capture every request, as it comes in, for a while. This may cause slight delays of minutes or hours, for the first few days after go-live.
Phase 2: We do not expect any downtime but there may be occasional, short periods of "offline for maintenance" during the transition and testing. If there are issues, we will roll back to the current storage solution while we troubleshoot and prepare to retry the deployment. In a worst case, we may end up clearing out the entire cache of captures repeatedly, until we get a bulletproof deployment and then let the system run full speed.
Phase 3: We do not expect any downtime, but there may be occasional, short periods of "offline for maintenance" during the transition..
GEEK-SPEAK
The long-term goal of all of this work is to allow us to greatly extend the retention policy at some point. This will be the first step towards growing our pre-captured screenshots of web pages to include every web page on the Internet. It is an ambitious, and admittedly impossible, goal because of the vast number of web pages and extraordinary growth of the Internet.
However, we have to start somewhere and achieving even 10% captures of the most trafficked web pages would mean less than 1% of requests would be considered new. As we approach that point, the need for capture generators will transition from new captures to just keeping requests refreshed.
The beauty of that lies in being easily quantifiable, which means that we can eliminate "New Requests" from our pricing model and confidently build the cost of generators into the service, overall. That will greatly simplify the sales process and will help to lower costs in the long run.
While these phases require a substantial initial investment and considerable increases to our operating expenses, we feel that this is a necessary evolution for our service. These added costs will be leveraged quickly, as our service continues to grow exponentially. So, we are betting on that continued growth. Eventually, the added costs will be a nominal portion of the overall operation and, with our loyal user's continued support, we believe that we will reach that goal and earn the freedom to give back to our valued customers!
English
Deutsch
Español
Français