{"id":50145,"date":"2018-09-06T10:50:00","date_gmt":"2018-09-06T15:50:00","guid":{"rendered":"https:\/\/blog.cpanel.com\/?p=50145"},"modified":"2018-09-06T10:50:00","modified_gmt":"2018-09-06T15:50:00","slug":"a-true-story-of-a-mission-impossible","status":"publish","type":"post","link":"https:\/\/devel.www.cpanel.net\/blog\/cpeople\/a-true-story-of-a-mission-impossible\/","title":{"rendered":"A true story of a mission impossible."},"content":{"rendered":"

This is a guest post from Tim Hollis, VP of Operations at JetApps! JetApps has returned this year to exhibit at the cPanel Conference, October 1st – 3rd in Houston, Texas. If you haven’t already, take a look at the agenda<\/a>, book your room<\/a>\u00a0(discounted rates apply until September 9th!), and get registered<\/a>!\u00a0<\/em><\/p>\n

\n

As a software company, nothing makes us happier here at JetApps than hearing stories of how JetBackup has significantly improved the backup and restore process for hosting providers. \u00a0Please enjoy the following true story of a JetBackup user who found a way to reduce his backup window by 50%!<\/span><\/p>\n

On A Mission…<\/h3>\n

We were on a mission to find a way to run daily backups of our entire server farm of 150 servers on to 1 single backup destination server. \u00a0Our ultimate goal was to complete this daunting task in a backup window of 7 hours or less between the time of 11pm to 6am to avoid disturbing our clients during business hours. \u00a0Before continuing this true story of a \u201cMission Impossible\u201d task that would eventually become reality, let\u2019s discuss the technical specs of our configuration: <\/span><\/p>\n

<\/a><\/p>\n

1st Run: Here We Go!<\/h3>\n

<\/a><\/p>\n

Our first backup job run time was not pretty, to say the least. \u00a0But without incremental kicking in to reduce the amount of data to backup up, it is hard to tell exactly where we stand. <\/span><\/p>\n

Conclusion:<\/b> We need 1 full incremental backup to better understand how long each server takes to run backups.<\/span><\/p>\n

2nd Run: First Incremental Backup<\/h3>\n

<\/a><\/p>\n

Great improvement from the first run but it is nowhere near our goal of 7 hours or less to fit inside our backup job window. \u00a0<\/span><\/p>\n

Conclusion:<\/b> There is a major IO & network bottleneck<\/b>. \u00a0Having all the servers run at the same time is clearly slowing down the backup process. \u00a0We decide to use JetBackup 3.1\u2019s push notification API call to send a remote command to a log server specifying the start and end times of each backup job on the server. \u00a0After waiting another 24 hours to collect the data, we did some calculations and decided we need to expand our backup window to 12 hours from 10pm to 10am.<\/span> \u00a0This allowed us to schedule 20 server connections to our destination server at a time, reducing the IO stress from our destination server.<\/span><\/p>\n

3rd Run: Improved Scheduling and Hours<\/h3>\n

<\/a><\/p>\n

Made quite a bit of progress reducing the total backup time per server. <\/span>But we still have work to do!<\/span><\/p>\n

Conclusion:<\/b> We need a closer look to see what exactly is happening in real time. \u00a0After running \u201ctop\u201d and \u201cps\u201d commands throughout the night we found some interesting results:<\/span><\/p>\n