{ height: 1%; } - Ruby on Rails and User Interface Design: Data Loss :(

Data Loss :(

Richard White — Fri, 29 Jun 2007 09:28:00 +0000

We passed two major milestones yesterday: our millionth SlimTimer hour was logged and we had our first ever widespread incident of data loss. Ugh!

What happened:

Due to what I’ll call ‘operator error’ we lost all data on any tags or tasks created after June 28th 04:00 GMT
We stopped the server to repair the database using nightly backups to fill in the gaps the best we could
We created a script that would create ‘recovery’ or dummy tasks to be placeholders for the tasks that we lost and give us something to attach your time entries to.

Who’s affected:

360 People had tasks that were lost and now have placeholder tasks, with names like ‘[Recovery 190]’, in their places
More people may have lost tasks if they were created yesterday but not linked to any time entries
All tags added yesterday to both tasks and time entries were lost.

What you need to do:

If you have any placeholder tasks you need to rename those and add any tags, coworkers or reporters that may be missing
Check over your entries from yesterday and re-add any tags that may have been lost.

If you’re one of those 360 people you’ll be getting an email from me shortly, but anyone should feel free to contact me with any concerns they have.

We’re working on making sure this doesn’t happen again and increasing our database backup regularity.

a million apologies..

Rich

"Data Loss :(" by raand

Mon, 12 Nov 2007 01:46:01 +0000

Thanks for timely notice and being completely honest. Good work on some of the data recovery as well

"Data Loss :(" by Chris Matthais

Tue, 03 Jul 2007 16:12:44 +0000

Well, really 360 appologies, right? I’m glad you sent me that email but I figured it out when I saw the recovered time in my reports. No harm done. Luckily for me, I was only working on one project that day and probably one task, so I wasn’t hit hard… but thanks for the post and the email to let me know.

"Data Loss :(" by mike@kkl.com.au

Mon, 02 Jul 2007 02:09:38 +0000

Bummer ‘bout the loss, but like the others have said, being straight up counts for a lot.

Let me know if you want any help / advice on backup for postgres – we’ve been using it for years & one of our guys is a committer.

"Data Loss :(" by Richard White

Sat, 30 Jun 2007 05:48:32 +0000

@Matt: Yeah, Lance has used that as well and we’re actively looking into using that in the future (we’re on Postgresql currently).

"Data Loss :(" by Matt Platte

Sat, 30 Jun 2007 01:03:47 +0000

I’ve used that MySQL binary log/automatic replicator mechanism save me from myself.

"Data Loss :(" by Alex

Fri, 29 Jun 2007 23:43:28 +0000

Thank you for letting us know so quickly! I have been using the service for months now, and never had any problem. This kind of things happen, we understand. Keep it up, we love what you are doing!

"Data Loss :(" by Arron

Fri, 29 Jun 2007 18:23:23 +0000

I’ve got to tell you, I just signed up yesterday to use the service and having it crash / loose data the same day I started using it…

But! The transparency is great—without it I just would have found another timing solution. I’ll be sticking with you a little while longer to see how things work out.

"Data Loss :(" by Jason Coleman

Fri, 29 Jun 2007 17:52:43 +0000

Thanks for the update. I feel with you. I know how stressful these situations can be.

Luckily I didn’t use slimtimer too much yesterday. Thanks for being honest with us.

"Data Loss :(" by Nelson

Fri, 29 Jun 2007 17:46:24 +0000

Ouch! But you have an awesome site and we, your users, love you. Thanks for the hard work of trying to recover!

"Data Loss :(" by Kristopher

Fri, 29 Jun 2007 17:23:55 +0000

That really sucks to lose that data, but props for the openness and getting it taken care of promptly. You guys rock.

"Data Loss :(" by DK

Fri, 29 Jun 2007 17:18:58 +0000

You’ve wasted at least 15 mins of my time, but since you’ve saved me at least 40+ hours over the past year, I guess I’ll have to let it slide! :)

Thanks for the info on what happened.

"Data Loss :(" by Mark

Fri, 29 Jun 2007 17:15:36 +0000

You guys rock. Total transparency…. builds trust

Does lead one to think about a slightly more automated way to do backups. It’s one thing to do CSV dumps on a monthly basis by hand, but maybe a cron job that emails the users the CSV file each week or something.

I’d Pay for that for sure!