More Servers

Rack Servers in CabinetOur baby Tangler is growing up (sniff). The beta is currently hosted on servers in Sydney, but load is forcing this to be expanded. We also want to shift hosting to the US sooner rather than later (where bandwidth is much faster and cheaper). So shiny new Opterons, tier 1 bandwidth, and a San Francisco data centre here we come.

The problem now is the management of this infrastructure is getting too large to handle without dedicated people, so we had to figure out how we’d handle the expansion. Do we continue to try to run it ourselves or expand the team with some systems administrators?

Saving money is always a good thing, so we considered scraping along with part-timers (Minh, Craig and I most likely) for the first few months, but we felt this would be risky. Why? Because managing any sizable system is a large, complicated job that can’t be done half-hearted. The backups will fail; the HDs will fill up; that tier 1 provider will drop your class C off their AS due to a “clerical error”; you’ll fail to install that kernel update and get hacked. There are a million things that absolutely will go wrong – and yes, all of the above have actually happened to me at various times in the past. We needed someone dedicated to the problem.

That leaves the second option: hire a dedicated systems administrator. But we face a bit of a scale issue there. You can’t hire just one; these little buggers breed! If you hire one, no matter how hard you try there will be 10 within 3 months. OK, not really, but seriously, try sharing a 24-hour pager between two people, let alone one. I didn’t want to hire a team of people. We also didn’t have time to find and hire the right people, setup systems from scratch and still meet our growth at the moment. I started to consider outsourcing options to bridge the gap between kick-starting and our own team. After a bit of scouting and talking to other startups I came across a better solution.

Enter BitPusher, a company that specializes in operating sophisticated infrastructure using a commercial model that makes it accessible and practical, and a lot cheaper than hiring your own team. They’re working with us to develop a spec on our new systems, and then they handle hardware procurement, deployment and operations while we go to the beach! All right, maybe not. But it sure is nice to have the problem taken care of by people dedicated to it. Thanks guys.

And after a little pushing I’m happy to see Daniel (BitPusher’s CEO) has started a blog.

4 comments so far

  1. Craig on

    I would pay any amount of money if it meant I didn’t have to carry a pager.

  2. Michael T. Halligan on

    Marty.. Just wanted to say Welcome Aboard to the BitPusher express!

    BTW, our network is somewhat better than mere Tier-1 providers. A tier-1 provider, by definition is an ISP that owns it’s own transport mechanism (fiber) and does not purchase any private peering. This approach ends up becoming rather limited in terms of it’s routing capability, given the # of isps who don’t do much public peering.

    Our main bandwidth provider is Internap. Internap’s model is not to build a routing infrastructure in each datacenter it’s in, and buy bandwidth from all of the providers at that datacenter.

    Internap then acquires a new AS number for that datacenter, and then uses some pretty sophisticated route optimization combined with BGP. The end result is that their routing ROCKS!

  3. Martin Wells on

    Even better… and yeah, great to be working with you guys.

  4. Scott Farquhar on


    You may also want to chat the the contegix guys. We use them for all our hosting, and they are incredible. Like having your own System Administrator.

    Send me an email if you want an introduction.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: