Paul Lindner

So had a four hour power outage last night. Pretty sure we're down to 2 nines of availability for the year.

You'd think with all the new Smart Meters that

Pacific Gas and Electric Company should accurately know about power outages in near real time. Instead of alerting on their usage graphs it appears that the primary feedback mechanism is phone calls.

I've asked for a callback. When I do I'll post it to the Postmortems community. Ought to be interesting...

#gplus

Just so you know the background here.

Just so you know the background here. Moving to the new fetcher triggered this. This also happens whenever we add a new datacenter. The NY Times staff figures this out pretty quickly and we have notified them of this issue.

http://b/8227621

Originally shared by Lauren Weinstein

(New York Times links are having increasing problems embedding into G+ reliably.)

http://www.nytimes.com/2013/02/26/sports/whistles-may-harm-referees-hearing-study-says.html?partner=...

#gplus

Is there an easy way to null-mock an RPC interface in Java/Apps-Framework? I couldn't figure it out with some time in code search

I'm seeing people using easymock and mockito to do this, but that feels wrong (RPCs don't return null!) and also means that these testing frameworks might find their way into production code if we're not careful.

Here's an example:

import com.google.ccc.groups.index.server.api.proto2api.CommonGroupsApi.CommonGroupsIndexSyncService;
...
@Mock
CommonGroupsIndexSyncService.ClientInterface mockSyncServer;
...
bind(CommonGroupsIndexSyncService.ClientInterface.class)
.toInstance(mockSyncServer);

Perhaps we could have a stub implementation that could do the following:

1) Implement returning defaultValue for all requests.
2) Implement returning RpcException for all requests.

Extra credit for the following wrapper class that would allow for injecting behavior into the pre-rpc/post-rpc phases:

public class ForwardingRpcClientInterface {
...
pubilc T delegate() {}

beforeRpc(...);
afterRpc(...);

public Object invoke(...) throws Throwable {
// defaults to forwarding to delegate
}
}

With something like this you could implement a client-side cache or short-circuit requests here. You could also implement the defaultValue/RpcException classes based on this.

#googplus

It's 2013 -- why do we still have problems with hotspots in Bigtable/Kansas?

I have not looked into the backend design, but this seems like something that's been solved in the past.

Do we not have
- Dynamic-sized replicas (add more read-only replicas for hot keys)?
- What about adaptive sharding/rebalancing (rehash based on usage)?
- Hierarchical storage? (automatically move dormant accounts to older/slower hardware, move back when reactivated)
- Or... the ability to manually move a single hot key to it's own instance. (variant of adaptive sharding above)

I know about the stubby cacher and how that can help, but that seems like treating the sympton rather than the source of pain.

#googplus

Paul Lindner

Only a week to go until the radio play of Neil Gaiman's Neverwhere on BBC Radio 4. It has quite the all star cast:

So had a four hour power outage last night. Pretty sure we're down to 2 nines of availability for the year.

No, this isn't today's doodle. It is some very creative high-speed liquid photography.

The grandfathers of Grunge are back on a new Sub Pop Records release 25 years after their first.

Hi fine folks at Montclair Vet clinic. You should merge your Google+ page with your local page here:

And here's the official word on the Azure outage.

#ifihadglass I could safely post more photos like this.

Just so you know the background here.

A birch pig overlooks the crowd at Plow.

Anyone going?

Yes, this is real.

Mesmerizing. Be sure to click on the live map.

Not much technical detail here other than "The Landing Page didn't work on IE"

I recommend that you check out the Loring Cafe. They're opening up from 5-8 during first friday.

There have been some historic photos posted, might be of interest..