Posterous/Story

From Archiveteam
Jump to: navigation, search

A short story of how Archive Team saved Posterous. The story has been dramatized for your enjoyment.

Prologue

How did Archive Team save Posterous? How did we save 10 million blogs before April 30th, 2013? Shh, it's a secret.

I'll tell you anyway :) .

Deus ex machina

It was normal night (for those in UTC-09:00 through UTC-04:00) in #preposterus when someone noticed a peculiar tweet to Archive Team. It was a screenshot by an employee watching Archive Team hammer Posterous on the Tracker's leaderboard and on the internal sysadmin side.

“Heh. Tell him we’ll turn it down if they stopped banning us,” joked someone.

“I wished you wouldn’t include me in the Twitter conversion,” said another annoyingly.

This mysterious Vincent, working on a late-night shift, was trying to keep Posterous up and running while Archive Team, trying to avoid those bans, kept hammering away on their servers.

“You better throttle down,” said Vincent on Twitter. “A few requests per seconds should surely be enough to save 2 million blogs.”

“It’s not 2 million. It’s 10 million!'” exclaimed someone while jumping in their seat.

“Well,” replied Vincent, “There’s other options. At this rate, your efforts will be in vain.”

Archive Team recoiled.

“Oh dear, I think we better cut him some slack. I hope we didn’t upset Vincent and his team.” A Tracker admin throttled down the rate limiting.

vincent (~TheAlmightyVincent@admin.posterous.com) has joined #preposterus

Vincent piped out, “Hello everyone. I do not mind the crawling, but might I suggest a scheme to lessen the load on our servers. Perhaps, some dynamic throttling would be satisfactory. We appreciate if there was no fixed rate.”

“What would be a reasonable level?” asked someone.

“Your current reduced rate is appropriate,” said Vincent. “But the problem occurs with our infrastructure. Like many websites, our website relies on efficient use of caching. However, when crawling occurs, our caches fault up.”

“Oh no!” quivered someone. “What shall we do?”

“You will need to monitor the latency,” commanded Vincent. “If the latency exceeds a reasonable threshold, you shall throttle your requests down.”

“What would be reasonable? 400ms?”

“Yes,” said Vincent.

“Hmm… That does not seem like a trivial rewrite to the Warrior infrastructure,” pondered someone. “Can Twitter just give us a data dump?”

“I cannot authorized that” boomed Vincent, “for I am only a lowly servant.”

Archive Team continued to ponder. “The issue we are having is the hourly bans.”

“Yes,” said Vincent, “after the move to Twitter, anti-abuse is strictly enforced.”

“Why?” someone peeped.

“To avoid situations like this,” replied Vincent.

Someone sighed. “We’ll never get this done in time.”

“Might I ask how you are crawling?” inquired Vincent.

“We’re using wget. We only want the web pages for archival purposes.”

“I see,” said Vincent. “I suppose you are getting hefty amounts of spam and pr0n.”

“I have an idea!” exclaimed someone. “Can you set up a server just for us?”

“Or whitelist some IP addresses!” suggested someone else.

“Or whitelist a User Agent!” cheered another.

“Yes,” said Vincent, “I can. I will dedicate a server for you, however, that will not be available in a few days.”

“All right!” rejoiced Archive Team. We continued to question, “What else can we do? Beer? Snacks?”

“A funeral for Posterous,” someone quipped.

“Not much else can be done, unfortunately,” said Vincent. He paused. “Perhaps, I can leave the service past the shutdown date.”

Archive Team goes silent.

“Yes,” said Vincent, “please keep this quiet. Keep it… a secret.”

“Lips sealed!” said everyone.

“Now,” said Vincent, “I wished I have not used my real name for the purposes of refuting this conversation in the face of my contractual agreement to my employer.”

Thus on the 4th of March of 2013, a secret agreement was forged between Vincent and Archive Team.

Bans don’t Stop the Archive Team

“Alight. How are we going to make this dynamic throttling happen?” began the Archive Team.

“Maybe we can modify the Seesaw client?”

“I think it might be easier on the Tracker side. The Tracker already has rate limiting. I think we can just do an HTTP request and measure that.”

Discussion and off-topic banter continued throughout the night.

“I JUST WANT HUGS!” popped SketchCow.

“Hey SketchCow, Vincent is an Posterous engineer!”

YOU CAN’T STOP US,” said SketchCow towards Vincent, “but I’m fine with us working together to make it orderly.”

“Thank you,” said Vincent. “I appreciate your cooperation.”

vincent has quit (Remote host closed the connection)

Archive Team continued on: munging the database, looking at WARC files, and the like.

“All the results coming back seem to be 0 MB,” chimed someone.

“We broke Posterous!” joked someone else.

Indeed, the Archive Team did break Posterous. But thankfully, Posterous was back online the next day.

“I’ve done some calculations based on the number requests per minute to the tracker,” said someone. “It seems like there’s about 1500 Seesaw clients running.” (That’s a lot of clients!)

While Archive Team awaited for the special server, the scraping and grabbing continued on. A script was made to adjust the rate limit and it was installed on the Tracker. A few members worried about the rate limit. Was it too aggressive? Or could we go faster?

A few days passed.

“I’m still getting banned,” complained someone. “Did anyone contact Vincent on Twitter?”

“Has our little engineer shown up?” asked SketchCow, “He should be here because we’re going to ruin his weekend. In the face.”

“I’ve got a reply from Vincent,” announced someone. “He says we should try our dynamic throttling. How should I reply? ‘We have already been using dynamic throttling. We’re just waiting to see what you have for us.’”

“I hope we won’t have to treat him like we treat Yahoo!” said another, “Posterous should better look out; I’ve got my internet connection ready!” (This paragraph was rephrased to avoid obscenity.)

vincent (~TheAlmightyVincent@admin.posterous.com) has joined #preposterus

“Hello all,” announced Vincent, “I believe things are going well? You are free to increase your rate limiting slightly.”

“Actually…” squeaked someone, “we were wondering if you lighten up on the IP address bans.”

“No can do,” declared Vincent. “Might I ask what User Agent is being employed?”

Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27” said someone meekly.

“How dare you!” bellowed Vincent. “Not properly identifying yourselves. You should be ashamed. Tsk-tsk! …Regardless, may I ask you a favor. Please use HTTPS and the string ‘ArchiveTeam’ within the User Agent.”

“Let me try,” volunteered someone. “There! How is that?”

“Thank you,” responded Vincent. “I can see your request. To make matters brief, your request was rerouted to a special server isolated from other clients. IP address bans are not enforced on this server.”

“Awesome!” Archive Team said in joy.

“I’m going to update the client scripts right away,” said another.

“So,” said Vincent, “you have your server. I remind you to be good internet citizens. Your actions are my responsibilities sitting on my shoulder.”

vincent has quit (Remote host closed the connection)

“Please, nobody put this conversation on a paste web site or anywhere for that matter. Someone did so for an earlier conversation and he got in trouble,” said someone remorsefully.

Archive Team maundered about the sanctity and secrecy of the agreement. Eventually, they focused on the rate limiting. It seemed like there was no need for the dynamic throttling anymore.

Because the tracker enforced the minimum versions of Warrior and Seesaw client scripts, it took some time before Warriors auto-updated and users to manually upgrade their scripts. During this time, some discussed whether it was a good idea to use the special User Agent.

vincent (~TheAlmightyVincent@local-isp.com) has joined #preposterus

“Hey Vincent,” said someone, “the server you set up for us seems kind of slow.”

“I estimate you are able to achieve 30 requests per second with that server,” replied Vincent. “Perhaps you should filter out spam blogs from your queue.”

“I hope Posterous isn’t slow because of us. There appears to be too many requests and a lot of unreturned claims,” replied a Tracker admin.

“I have fixed an issue in the routing,” announced Vincent. “You should see better responses now.”

vincentch has quit (Remote host closed the connection)

Archive Team spent time re-prioritizing the queue. Banned accounts were filtered out and put aside for later.

vincent (~TheAlmightyVincent@local-isp.com) has joined #preposterus

“Hello,” said Vincent. “Since your requests are segregated to that server, the maximum number of Unicorn workers have been saturated. As a result, your connections are being held at the frontend.”

“That’s good for Posterous,” replied someone, “but we can only do 5 users per minute now. Our boss, SketchCow, won’t be happy.”

“You asked for a box”, said Vincent feeling taken back, “and a box I gave you.”

vincentch has quit (Remote host closed the connection)

Archive Team continued their discussion.

“I think we can avoid scraping /login. It a significant overhead.”

Some talks about the number of claims and requests continued. Rate limiting settings were fiddled. Algorithms were tweaked. Code was committed.

“Hey,” said someone, “did you remember to use HTTP?”

“What?” questioned another.

“I mean HTTPS. Vincent said to use HTTPS.”

“HTTPS? I think he meant HTTP. The HTTP path is faster.”

“Yeah, but, explain why Vincent would want us to use HTTPS.”

“Someone should direct message Vincent on Twitter.”

“I think we should be using HTTP only,” complained someone. “The Wayback Machine makes a distinction. Otherwise, if users enter an address directly, they might not get a result.”

The discussion about HTTP versus HTTPS continued. Thoughts about the rate limiting dribbled throughout.

“I did some calculations,” announced someone, “and they’re on the wiki now.”

“We need more speed,” said another.

“I’m still getting rate limited.”

“Posterous is down.”

“Posterous is up for me.”

“The Tracker is filled with 0MB results.”

“I think we should just remove the stupid User Agent.”

“I did calculations. We can’t make it.”

“We need them to keep the machines open,” declared SketchCow. “I’m going to have to openly embarrass them.”

“I think we need a two-pronged approach here,” suggested someone. “One with the Archive Team User Agent and one without it.”

“I think I found a bug,” said another. “Not all blogs are on the Posterous domain.”

More discussions about the User Agent continued. Some decided to manually edit the client scripts to use the Firefox User Agent. Some decided it was time for a “call to action.” We needed many clients running on many IP addresses without the special User Agent.

“The problem,” said someone, “is not that we do not have enough people downloading Posterous. The problem is that they do not want us to. And it’s not because they hate our efforts. It’s because their servers cannot handle it.”

The ideas of User Agents, IP addresses, bans, the Tracker, rate limiting, the Warrior, virtual machines, FAQs on the wiki page, Seesaw pipeline clients, Linux, Hacker News, EC2 instances, Digital Ocean, grep, Varnish, root, GitHub, continued throughout. Plenty of users arrived into the IRC channel. During that time, days passed by.

Tired, Hungry, and Irritated

vincent (~TheAlmightyVincent@local-isp.com) has joined #preposterus

“How are things going?” asked Vincent.

“Fine. I guess,” answered someone.

“In a few days,” said Vincent, “I may deploy another server.”

“Oh, good. Bro, we can’t get close to done at this rate. You should be embarrassed that your export function has trouble finishing before the shut down date.”

“We will get there,” said Vincent. “We have obligation to maintain the functionality of the server. Serving our users is our top priority.”

“Dude. That’s not what I said.”

“I fail to understand why I should be embarrassed,” said Vincent.

“Look, if all your users were to hit the export button the day of the shut down announcement, will they be done exporting by the shut down?”

“Probably,” responded Vincent. “They enter a queue and is processed offline by job servers.”

“‘Probably,’” was said in a mocking tone. “Now that’s is what I like to hear. If you can’t tell, I’m upset.”

“I fail to understand the hate,” asked Vincent.

“I’m tired, hungry, and irritated.”

“We and the Posterous team,” began Vincent, “have been working dutifully for a very long time. We have been paged night after night to keep the web site up and running every day.”

“Bro, don’t take this personally.”

“Honestly,” said Vincent, “it is hard to not take it personally. To work on something for a very long periods of time, expending great efforts, and spending late evenings and early mornings is difficult to not take it personally.”

“Well. Are you agitating internally to keep it going?”

“I am trying to work with you this very minute,” said Vincent. “I have been keeping the servers online and working together to ensure Archive Team meets its goals.”

“But every time Archive Team has to take action, it’s because the owners have betrayed their users’s trust.”

“I do not understand why have ‘betrayed’ our users' trust,” stated Vincent. “We maintained Posterous as long as possible. We don’t own the data. They belong to the user. They always have the option to pull their data out at anytime.”

“What about the dead users? What about those who’ve forgotten and only to come back six months later? They will be like ‘Fuck!, where did all my shit go?’” Shrugging, “You didn’t make this decision…”

“We have made it possible to pull their data out,” continued Vincent. “We made our best effort to contact them and provided them months of time. I do not know what else we can do for them. We do not have the ability serve legal notices to their homes requesting they remove their data.”

“Hmm.”

“Unfortunately,” said Vincent, “these things do not function by themselves for free. It requires time, money, and effort.”

“Whatever. How’s your end doing.”

“Good,” replied Vincent. “Another server has been added.”

“Are these new boxes or just existing ones?”

“Existing ones,” said Vincent. “It looks like utilization is now dropping. You may throttle up.”

vincent has quit (Remote host closed the connection)

“Dude, don’t take it out on Vincent,” cautioned someone. “He’s the only one that’s helping.”

And so, Archive Team continued on. It appeared that the second server helped.

We were Nervous

A few days later: vincent (~TheAlmightyVincent@local-isp.com) has joined #preposterus

“Hello,” greeted Vincent. “How are things going?”

“I think we’ve maxed out the other server,” responded someone.

“Hmm. I need to get a sense of your rate,” said Vincent. “I want to know if you are on schedule as I need to balance our ability to maintain uptime and between your goals.”

“We’ve calculated that we need 6 accounts per second excluding any spam blogs,” answered someone.

“I see,” said Vincent. “I am analyzing your graph and I can see that, with a third server, you will accomplish your goal within 42 days. That happens to coincide with the shutdown date.”

“Our current rate is 0.6 users per second,” reported another.

vincent has quit (Remote host closed the connection)

That did not sound good. Only 0.6 users per second?

vincent (~TheAlmightyVincent@local-isp.com) has joined #preposterus

“One server added,” declared Vincent.

vincent has quit (Remote host closed the connection)

At this point in time, it was the 20th of March. Business progressed as usual. Users continually entered and left the channel. Some attention was diverted to another Yahoo! project.

“Vincent just direct messaged me on Twitter. He is asking whether we are hitting Posterous without the special User Agent as Posterous is having trouble,” said someone. “Since when was I the primary contact for the project?”

“Posterous is down.”

“Nah, it’s up. They’re just slow as usual.”

vincent (~TheAlmightyVincent@local-isp.com) has joined #preposterus

“Hello all,” said Vincent. “How are things going?”

“Rather well,” replied someone. “We’ve been distracted by Yahoo! Messages.”

“Are you receiving enough throughput?” inquired Vincent. “How is your progress?”

“I think we will be able to do it based on my maths.”

“Not by my math,” another disagreed. “Every time I redo it, it shows we are slowing down.”

vincent has quit (Remote host closed the connection)

The deadline slowly approached Archive Team. The channel was fairly silent. Only users were joining and departing.

Item riccamandoosa.posterous.com: Step 3 of 7 - Downloaded 16760 URLs,” moaned someone. “Oh. My. God. Make it stop already!”

Downloading the internet is serious business.

“I did some math,” said someone. “We are slower than before, again.”

“Drinking has commenced,” announced someone. “This may get messy. Time to fire the cannons… Tonight we dine on Posterous!

(But wait a minute, all this data had to go somewhere. It did not go directly to the Internet Archive. Instead, it went to a staging server to be prepped by a Megawarc Factory. Unfortunately, the disk was full. 8 TB of data. But that was SketchCow’s responsibility. And of course, he fixed it.)

“It’s April 6th,” began someone. “3,677,720 more accounts to go and we are not even half way done.”

Much virtual crying occurred. More calculations was done. Days went by. Archive Team continued.

Formspring announced they were shutting down. The default project was switched from Posterous to Formspring. It’s now the 15th of April and there are only 15 days remaining.

“How many more blogs do we have left?” asked someone.

“2 million” responded another.

Archive Team has not heard from Vincent for a while. Posterous has become slow. We became nervous. We turned the dial to 11.

“It doesn’t look like we’re going to finish.”

“I doubt it’s going to just shut off on the 30th.”

“I think it will.”

Only 5 more days remained.

“Ha ha,” began someone. “This entire block of IP addresses I have is blocked from Posterous.”

“The tracker is not moving at all,” complained a Tracker admin. “Why is the limit set at 10?”

“We broke their banlists again,” responded someone. “We need to fix our dynamic throttle scripts to check for bans.”

“Look at all this red!” said another. “Wget keeps failing.”

“Still banned here.”

“I’m banned as well. Fun times.”

“The tracker hasn’t moved in a while.”

“Everyone is banned.”

“Does the Tracker allow results from Tor?”

Topic for #preposterus is: DON'T YOU JUST LOVE BLOCKING PEOPLE DUPLICATING DATA FOR A HISTORICAL ARCHIVE?

“Today’s average is only 0.13 items per second,” gasped someone.

“They still have that ban list overflow problem,” suggested someone. “Just try to get insta-banned and cycle the IP address. It will push out the older IP addresses from the list allowing others to continue.”

“Alight,” said someone. “I have a script that chugs away at Posterous. It checks every 5 minutes to see if I’m blocked. If so, it reboots my modem.”

“Madness!” declared another.

“Would it be possible for someone to ask Vincent to see what’s going on?”

“Someone could direct message him on Twitter,” someone replied, “but that might lead him into trouble.”

“Well, it was just an idea. I’m not sure how legitimate are these servers configured for us. Based on what I’m seeing, we aren’t going to make the deadline. It would be ideal for them to just turn off public access on the 30th and leave it up for just for us.”

Archive Team was still banned and not much could be done.

“Hey,” chimed someone, “could a Tracker admin allocate me blog.cloudflare.com to me, please?”

“Done,” responded a Tracker admin.

“Gasp,” gasped someone. “How did you do that? Is there a separate queue?”

“We haven’t even begun the list of non-posterous.com blogs at all,” responded another.

Only two days remained.

“This is insane,” exclaimed someone. “I think they might be blocking all of EC2 now!”

“Huzzah,” someone said joyfully, “I found a way to make my modem get a new IP address without needing to reboot it. Now my family won’t complain whenever they lose network connectivity.”

“Are the ban lists even rolling over anymore?”

“Not as far as I can tell.”

Streetfiles was shutting down. Attention became diverted.

“Well,” said someone, “it’s been a good run.”

“It was excellent,” said SketchCow. “I am going to make them very embarrassed.”

The Deal

vincent (~TheAlmightyVincent@admin1.posterous.com) has joined #preposterus

“So Vincent just told me this,” began someone. “They’re keeping the site up for a month as planned. But what’s not(!) part of the shutdown plan is to allow specific User Agent access after tomorrow. Guaranteed 30 days. Probably more. Ohai Vincent.”

vincent has quit (Quit: leaving)

“So he asks for some discretion, e.g., not Tweeting about it. In exchange, we get a month to bang on their infrastructure. No bans. Using the User Agent will bypass the Sunset page. The code is already implemented.”

“LOL. Woo.”

“Wow.”

“Damn,” said someone, “I wished they could have told us when they re-added the bans.”

vincent (~TheAlmightyVincent@admin.posterous.com) has joined #preposterus

“IP address rate limits have been removed for Archive Team User Agents,” declared Vincent. “I am purging the ban lists. Clients that do not identify themselves as Archive Team are subjected to the ban lists as usual.”

“Excellent,” said Archive Team.

“Here is the situation,” explained Vincent, “Our servers are not an infinitely scalable sink of requests. Archive Team is getting one third of the resources but too many people were attacking the other remaining resources.”

“We panicked. As the deadline approached, we configured the Tracker to hand out the User Agent at a 50 percent chance,” said someone.

“You should have contacted me,” replied Vincent. “Whenever someone does something like this, it is usually my responsibility. Even if it is 3 AM in the morning. However, I see by the leaderboard that the Tracker is flowing again.”

“Hey, speaking of the Tracker, where is everyone? We’ve got to configure the Tracker to use only the special User Agent.”

“The special User Agent is now back for everyone now,” announced a Tracker admin.

“Ban list cleared,” declared Vincent. “I am seeing many requests for /login however.”

“We’ve already fixed that, but some links redirect to /login and we can’t check for that.”

“Hmm,” pondered Vincent, “I suppose that is a moot point anyway. After the deadline, I am going to open the floodgates to all Archive Team clients. Of course, please do not hammer the infrastructure; users are still exporting their data.”

“Vincent,” said SketchCow, “in the interest of keeping things under wraps, our loose band of brigands wandering in now will be uninformed that a deal has been struck.”

“Thank you Vincent,” said someone.

The deadline, May 1st, approached.

“Well, it looks like a lot has happened since the last time I checked IRC. Posterous is still up?” asked someone.

“We have special access.”

“Two thirds of the servers are now serving clients with the special User Agent,” declared Vincent.

Work began on loading items onto the Tracker. The Seesaw and Tracker code was double checked to ensure the correct User Agent was used. Scripts were tweaked. Archive Team resumed is job.

“The non-posterous.com user list returns non-existent domain errors for several I tried,” said someone.

“Well,” replied someone, “we know the Posterous IP addresses. We can just do requests with the Host HTTP header.”

A few days passed quietly.

“This is a reminder to not talk about the project on social media. One person already has,” said someone. “I am upset because this is not the first time.”

“Name and shame them!” responded someone.

“It won’t work on volunteers. Save your enthusiasm until later.”

“Ha ha,” laughed SketchCow. “You are so adorably hot-headed. It’s fine. Just some guy who came late to the party. He didn’t know. Let the chatter happen but we’ll do our best.”

“I don’t see how this is a secret,” complained another. “There’s an IRC channel and news articles dedicated to this project.”

(Shhh…)

Magic

“There’s something wrong with getfile*.posterous.com,” said someone. “The images are not being downloaded.”

“Argh!” pouted another. “This large item just died.”

“So all the recent downloads were broken?”

“Yeah.”

“Vincent,” said SketchCow, “wake up please.”

PING” stated Vincent like a robot.

“I got a direct message on Twitter from Vincent,” said someone. “He says it’s now fixed and we just go ahead.”

“How is the rate?” asked Vincent. “It appears that there is sufficient time to archive the remaining items. 80% of Posterous front-end servers are now dedicated to the Archive Team.”

“I feel like we should deliver Vincent a cake and high fives.”

“Are you hitting the RSS feeds as well?” asked Vincent. “It is redundant and slowing your progress.”

“I think that shouldn’t be a problem,” answered someone. “But only if you can block it with a proper HTTP error code. 404 should be best.”

“Cool,” said Vincent.

For the remaining days, Archive Team kept their course straight. Some talk about bandwidth and speed occurred. Archive Team's servers filled up a few times. The number of items in the queue slowly decreased.

It was now May 21st, 2013.

“So,” someone began, “it looks like it’s time to deliver a cake to Vincent soon?”

“Vincent, what cake do you like?”

“No cake required,” said Vincent. “But thanks.”

“What about a big juicy steak?”

“Favors are paid back in pizza and beer,” said someone.

“How about you buy a few Twitter Promoted Tweets?” joked Vincent.

Someone began looking for Vincent’s home address.

“This channel has taken a turn for the creepy,” remarked someone.

“The non-posterous.com sites seem to be working,” said another.

“100,000 left to do.”

Almost there!

“How come there’s people downloading Posterous?” asked someone. “I’ve tried to check a website on my browser but it just says Posterous is no longer available.”

“Magic,” answered Archive Team.

“I’m surprised we’ve gotten more than 90% of Posterous,” said someone.

“8000 items left,” began another. “5000 left… 315… 0…”

“I’ve got thousands of jobs running still,” said someone. “They’re taking ages to download.”

“I’m going to recycle the item claims,” reported one of the Tracker admins. “Still lots of small ones.”

“Holy Archivos!” exclaimed Vincent. “Only 278 to go.”

Almost there (again)!

“A huge one,” noticed someone, “just uploaded: 12930 MB.”

“Yow!”

It was May 30th, 2013.

Cake delivered,” announced SketchCow. “Difficulty Level: Not even mentioned once in this channel.”

“Nice!”

“Awesome!”

“Nice one!”

“Ha-ha-ha, epic!”

“We actually got Vincent a cake. That’s fantastic.”

“In the end, there will be cake.”

“Totally not a trap.”

“The cake is a lie.”

Almost there. Just only 1573 items left.

“Recently finished,” said someone. “41523 MB. Another big one: 30679 MB.”

“LOL. It’s crazy. We might actually finish.”

PING,” queried Vincent. “How y’all doing?”

“The last few are taking a long time because of the size,” answered someone.

“No worries,” said Vincent. “I’ll try and keep the site up for a bit longer.”

“Another one: 51857 MB. Now that’s some huge data there,” said someone.

“Well,” began another, “we’re nearing the end. Let’s reminisce about where we were when we heard about Posterous’ closure.”

One Forgotten Server

June 5th, 2013.

“Posterous is still up?” questioned someone. “Vincent must be performing some kind of voodoo to keep the servers up.”

“600 out now.”

“If only there was someone with Google Fiber that could help,” someone interjected.

“Google Fruit and Fiber! So the intertubes don’t get constipated!” joked someone.

Another pondered, “I just picture one forgotten server sitting in the back corner of their data center.”

“Vincent’s laptop,” replied someone.

“Huh,” said someone, “my items are receiving error 503 for everything. Ooh. The getfile*.posterous.com domains are returning errors.”

It was June 12th and only 477 items remained.

“Another 10000 blogs were discovered,” announced someone. “Anyone want to import the list?”

“I hope this doesn’t blow up,” said a Tracker admin.

“Wow,” said someone, “lot’s of users with 0 and 1 MB.”

“And… 10,000 more items done within hours,” confirmed another.

“I do want to know why so many of the jobs seem to fail,” asked someone. “Were they just too big?”

“Yes. Ran out of memory on the Warriors.”

“I doubt we can rely on Posterous being up for much longer now,” began someone. “Vincent has been enormously generous with his time.”

“‘Enormously generous’ is in my opinion, an inadequate description. I believe ‘above and beyond what is considered reasonable’ is more accurate.”

“Hi,” said a Tracker admin, “I added 600,000 more URLs discovered from the Wayback Machine.”

“Fire up the Warriors!”

“I think the Tracker is down.”

“The Linode dashboard shows 600% CPU usage and the console is showing lots of scary-looking messages,” said the Tracker admin.

“The items are failing because of the http and trailing slashes.”

“I’m fixing it now,” responded a Tracker admin. “All items are being reloaded with duplicates removed.”

“Are we still making progress?” asked someone.

“I can’t believe I’m saying this, but yes, we are.”

The Final 11

June 18, 2013.

“Only 12 left? How?” questioned someone. “Oh, we’ve got 11 now.”

“Release the claims and we get them done,” said someone.

“I’m already doing that,” replied an admin.

“What are those 11 remaining ones? Can someone verify that they are not spam?”

“Everyone of those are legit.”

“And huge?”

“Yep.”

Not much activity occurred. It was now June 24.

“I don’t think anything is going on here,” stated someone.

“Pretty much dead now. I think we should do manual grabs for the last 11.”

A few members did manual grabs.

“These aren’t that big at all,” said someone.

June 29: Topic for #preposterus is: WE HAVE EVERYTHING. MUCH LOVE TO ALL WARRIORS <3. Good night and turn off the lights when you leave!

We did it. We actually did it. We saved Posterous.

And although Vincent’s cake may have been sweet (and perhaps, delicious and moist), Archive Team’s victory may be the sweetest of all.

Epilogue

July 8, 2013.

“Ping. Anyone around?” asked Vincent.

“Your concern is fixed,” said SketchCow. “Enjoy the cake!”

“Thanks,” said Vincent.

“And it has been a great 2013,” said SketchCow.

So long and thanks for all the fish.—www.posterous.com/bye.html