Skip to main content
 

Finally solved a really weird network issue on my home network.

Multiple interfaces with different firewall rules on the same wire cause great confusion, but a few sysctl entries fix it.

https://netbeez.net/blog/avoiding-arp-flux-in-multi-interface-linux-hosts/

 

Early Google - Linux restricts

I found my first email to Google. From the year 2000. Since then I've worked through five companies, but Joan Braddi is still here.

And since I'm on the inside now I was able to find the that cl referenced the work I was doing at the time, which was adding search to www.redhat.com. (Or Red hat, as you can see here)

https://critique.corp.google.com/#review/8802/depot/eng/clientwork/CompletedSites.html

I do have to wonder if Red Hat's slow payment of their 5k+ contract fees hurt the early Google, or what might have happened had Google gotten into the RPM search field as I suggested to them way back when...

And for a real blast from the past see the search stats that were emailed out monthly(!)

https://groups.google.com/a/google.com/d/msg/partner-reports/XiVcELeVONc/AtcwCeBS7uYJ

 

Back in 1999 I worked with keen.com to provide a 1-900 service for linux support.  Today?  Well, just look at the snippet below..


http://www.keen.com/






 

Since you missed this before...

Since you missed this before...

Originally shared by Theodore Ts'o

Phoronix, alas, has perpetrated another example of irresponsible journalism.   I won't dignify said article with a web link, since I don't want to reward them with more ad hits.  So I'll link to the original Ubuntu Launchpad report, and include the comment I just made there:

Those specific fsck corrections --- fixing the number of free blocks and the number of free inodes --- is completely normal and is purely a cosmetic issue. There is nothing to worry about here.

What is going on is that ext4 no longer updates the superblock after every block and inode allocation; that causes a wasteful write cycle to the superblock at every single journal commit, and it also is a SMP scalability bottleneck for larger servers (i.e., with 32 or 64 CPU's). To fix this, we no longer update these values in the superblock every time we allocate a block or an inode. Instead, we only update these values when we unmount the file system, mainly for cosmetic purposes so that dumpe2fs shoes the correct number of free inodes and blocks, and at mount time we calculate the total number of free blocks and inodes in the file system by summing the the free blocks/inodes statistics for each block group. So in fact, ext4 does not depend on the correctness of the values in the superblock, but it does try to update them on a clean unmount.

In e2fsprogs commit id 2788cc879bbe6, which is in e2fsprogs 1.42. 3 and newer, we changed things so that e2fsck -n would not display this as something "wrong". However, we still do show this as something that we "fix" when running e2fsck -y or -p, since in fact it is a change to the file systems. See: http://git.kernel.org/?p=fs/ext2/e2fsprogs.git;a=commit;h=2788cc879bbe667d28277e1d660b7e56514e5b30

No one else has complained or noticed up until now, because other distro's apparently are capable of doing a clean shutdown allowing the file system to be unmounted cleanly. Ubuntu, unfortunately, is incapable of reliably doing a clean shutdown even when users request it, which is why Ubuntu users are seeing this behavior much more frequently, and apparently some people have panicked as a result. Sigh....

----

I will say that it is extremely irresponsible of Phoronix to make a big deal about this this before giving anyone knowledgeable (which unfortunately  does not include any Ubuntu kernel engineers, since as far as I know they don't have any file system specialists on staff) to comment on the bug.  No one from Phoronix even bothered to contact me to tell me they were posting this story, or to ask me for a comment.  I had to find out about it when someone asked me to comment on Google+.

However, from the perspective of trying to send as many ad clicks as possible to their web site, they are doing a heckuva job....

 

Please welcome Baron von Luxxury to Google+!

Please welcome Baron von Luxxury to Google+! Add him to your music or DJ circles today and check out his new album which you can snag for free for one week only. (though it's definitely worth paying for..)

Dubbed the "Post-Post-Synthpop King" by the LA Weekly, Baron Von Luxxury seeks to erase the lines between high and low art by merging elements of innovative dance music with the idiot immediacy of classic pop.

I had the good fortune to work with him at Red Hat before he became famous. Some day I'll tell some stories about hairnet banner ads and psychic web-sites trying to go legit by offering linux phone support. (yes! really!)

Originally shared by Baron von Luxxury

Happy Day After Valentines Day, My Darlings!

Please download my new album The Last Seduction. For Free. It just came out today. And for 1 week only you can download it for any price you want, including $0. I honestly don't care how much or even if you pay, I just want you to have the music! Please. Just take it.

http://baronvonluxxury.bandcamp.com

 

Until everything in the world is in the cloud this works well for getting the bits from the linux file server in the...

Until everything in the world is in the cloud this works well for getting the bits from the linux file server in the garage to the Google TV in the living room.

Plus it uses freebase.com data to enhance your content in new and interesting ways.

 

Social 2012

Here are my new years thoughts on the social web platforms. I've been pondering what the world can do to get ourselves out of this one man/one graph/one api situation we find ourselves in and what Google can do to help.

Facebook the product and Facebook Connect/APIs have sucked up most of the social web oxygen. Open efforts have largely failed -- the Google-led consortium OpenSocial has been relegated to niche usage like enterprise or contextual gadgets, OpenID use is declining, Webfinger never took off, oEmbed has murky IP, even RSS and Atom are use is declining.

In particular OpenSocial is now paralleling a different OSF -- the _Open Software Foundation_. Remember that? DEC/IBM/etal banded together to fight off a common enemy (Solaris/AT&T or Windows). There was some decent output from it (I loved me some Tru64 OSF/1 Unix) but in the end it was Linux that disrupted and became the server standard while Windows claimed the defacto client standard. Today both OSFs are in decline and don't define the market.
So what to do? Here are a few of my ideas, what about yours?

* Obviously getting market share for Google+ the product and Google+ the platform helps, it provides an alternative. However if we're not careful we end up with a Coke/Pepsi duopoly, since much of our growth will come at the expense of the wider ecosystem before it starts to take from Facebook.

* Try to build on open standards where it sees broad based usage. OAuth 2.0 is something that everyone (including FB) has actually implemented. Activity Streams and schema.org are ascending. Add social to these where it makes sense.

* Try to nurture the next disruptor and be prepared to jump on it when it comes. Any technology that Google promotes as "open" will likely meet the similar fate as OpenSocial. (And I hope schema.org is the exception here...)

* Do something about the Terms of Service encumbered internet that's slowly taking off.

Hopefully sometime in 2012 there can be a way for everyone to work together on social. I hope to live to see the day that Facebook, Google, Twitter, LinkedIn and a hundeds of smaller players can do something that lifts all our boats and benefits users.

 

Hey look, it's the Fedora Project! On Google+

Hey look, it's the Fedora Project! On Google+

Check out Fedora 16 if you're into Linux distros. I just finished upgrading the home server last night and it's running smooth. (even if systemctl reminds me way too much of solaris SMF)

Fun fact: I've been upgrading the same system image since Red Hat 6.0. No reinstalls.. and since I never had a CD/dvd all upgrades were via rpm/yum.

 

Couldn't agree more...

Couldn't agree more...

Originally shared by Brad Templeton

When I saw the news about Steve Jobs it was a shock. But the news that Dennis Ritchie has died was much more like a blow. I didn't know him super well, we attended Usenix together for many years, but he was mostly a quiet man, not the sort to seek the limelight.

Dennis was instrumental in two things -- the C language and Unix -- which are the foundation of almost all the computing in the world today. Certainly much of C came from its predecessors like B and BCPL, and many things in Unix from its predecessors like Multics, but these two focal points were where all of 60s and early 70s computer science came together, and spread out. Of course Linux and MacOS and Android are very direct descendants of Unix, but even Windows is filled with ideas from Unix.

Far more than Steve Jobs, pretty much all your computing is based on Dennis' work. Last week Google noted Jobs at the bottom of the home page, a very rare honour. But Dennis Ritchie would be even more deserving of it.

 

Interesting data on scaling opensource memcache codebase to 850k qps using the standard linux tcp stack on 10G ethernet:

http://www.cl.cam.ac.uk/~rss39/mm_comp.pdf

I wonder how our internal version of memcache stacks up? [moma search for memslap had no results..]


http://www.cl.cam.ac.uk/~rss39/mm_comp.pdf






 

who knew that my first device to run linux 3.0 would be my Nexus S!

However I'd not recommend installing the canary build of ICS quite yet.. It's hung on me a few times.









 

Fedora 12, Dracut, dmraid, mdadm, oh my!

It appears that Fedora 12 moved to a new boot init system called dracut.  Sadly due to a number of odd circumstances this has caused me much pain.  Here's my basic config

 
  • /boot and /  on /dev/sda
  • /var and /home on a partitioned software raid on /dev/sd{cd}
 
After an yum-based upgrade to Fedora 12 I rebooted.  We get to the point where we initialize the software raid and boom.  failure.  I'd seen this before, partitioned raid has always had some trouble in fedora.  Previously I had to modify the rc.sysinit script to reset the raid partitions, so I tried that again, moving that init to later in the boot sequence.  Reboot and yes, it works..
 
However then I noticed some odd things.  I was only getting a single drive in my mirrored RAID.  Further investigation revealed that I had a device dm-1 instead of sdc or sdd listed in /proc/mdstat...  Uh oh..
 
Looking more closely it appears that my drives were getting set up by dmraid as a fake-raid mirror:  
 

# dmraid -r 
/dev/sdd: sil, "sil_aiabafajfgba", mirror, ok, 488395120 sectors, data@ 0
/dev/sdc: sil, "sil_aiabafajfgba", mirror, ok, 488395120 sectors, data@ 0

 
I tried adding the nodmraid option to grub.conf but then the new dracut system started an infinite spew of messages generated by this mdadm error message string (lifted from Assemble.c)
 

fprintf(stderr, Name ": WARNING %s and %s appear"
" to have very similar superblocks.\n"
" If they are really different, "
"please --zero the superblock on one\n"
" If they are the same or overlap,"
" please remove one from %s.\n",
devices[best[i]].devname, devname,
inargv ? "the list" :
"the\n DEVICE list in mdadm.conf"

 

Drats! the mirrored fake raid had already mangled my second drive by duplicating the superblock!  Plus since all this was going on in dracut I couldn't fix it.  So I removed the nodmraid option in grub during boot and dug a little deeper. I found that I could keep dracut from doing all this nonsense by adding the following kernel options:

 

rd_NO_MD rd_NO_DM nodmraid

This allows for a minimal boot without dmraid or mdadm.  After that I was dropped into single user mode with the dupe superblock message.  To fix this required zeroing the superblock of sdd
 

mdadm --zero-superblock /dev/sdd1

 

And then rebooting (again!)

 
Once past this things started working somewhat normally.  To get my raid mirrored again I did the normal thing:
 

# mdadm --manage /dev/md_d0 --add /dev/sdd1

 

To get rid of the false-positive fake raid setup I found that you can do this with the dmraid tool itself:

 

[root@mirth ~]# dmraid -E -r /dev/sdd

Do you really want to erase "sil" ondisk metadata on /dev/sdd ? [y/n] :y

[root@mirth ~]# dmraid -E -r /dev/sdc

Do you really want to erase "sil" ondisk metadata on /dev/sdc ? [y/n] :y

 
The really odd thing about this whole incident is that I never had these drives in a fake raid setup before. 
 
In any case, hope this helps the few other people who might have this same problem.
 

 

Luxxury - "Sweet and Vicious"

Baron von Luxxury is an old co-worker of mine from the Red Hat days selling ads for redhat.com.  Check out the cute kitty in the youtube vid above.

I can't forget the time we accidentally ran run-of-network ads and you found banner ads for Dennys and some online Hair-Net website running next to Linux content.

Or the wonderful partnership with Keen.com to provide amateur linux support over the phone.  I hear that they're mostly psychics and phone sex these days.

In any case check out Luxxury when you can get the chance.  They play all over the Bay Area at times.

 

All Good Things...

I can't believe it myself, Paul leaving Six Apart? Yes, it's true -- I've accepted an offer to be Hi5's Architect.

You can't fathom how much I will miss Six Apart -- the smart people, the positive dynamic environment, the technology and vision. I'm proud of what we've accomplished together and hope that my contributions have a lasting legacy beyond my final day, October 20th.

It sure has been a wild ride. Hard to believe it, but when I arrived TypePad was in the terrible twos - 2 app servers, 2 web servers, 2 Postgres 7.3beta databases, and (oh my) 2 SnapServers. Linux Kernel 2.6.5 had just been released and we were pushing a little over 10Mbps of traffic. Since then we've scaled up beyond belief while overcoming all sorts of obstacles. We've seen many new Typepad Stacks, two colo moves, three new ISPs, three new versions of Postgres, and dozens of software pushes all while building up great Engineering and Ops organizations to care and tend it all.

My immediate plans are to wrap up all of the loose ends while pushing forward on one last hurrah -- finishing up a simple version of Typepad and Mogile integration. If there's unfinished business or you want a brain dump of anything please let me know so we can take care of this in the next week and a half.

Don't forget to join me for a very special FooBar on October 20th to reminisce about all the great success we've shared over the past two and a half years.

After that... well, let's try to keep in touch.

I'll do my best to blog more on my Vox Blog, my LiveJournal, and my new experimental Typepad Blog. You can email me at lindner@inuus.com.

You can also find me at the following places



Thanks again for everything!

Paul