Find and replace in files on linux/unix

say you want to replace all occurrences of “Batman” with “Superman” in a bunch of files in the /home/nathan/superheros directory:

find /home/nathan/superheros -type f | xargs perl -pi -e 's/Batman/Superman/g'

this works by listing all the files starting in the passed sub directory find /starting/in/this/directory -type f and passing the file names to perl using xargs

remember finding -pi -e is delicious.

a pithy guide to railsconf 2012

Some one liners from what I thought were the most relevant, most novel, and most provocative talks at railsconf 2012.

Progress – David Heinemeier Hansson @dhh

• Progress is good, but not always at first. e.g. OSX scrolling direction.
• What is our attitude towards change: Curious or Suspicious?
• “A conservative is a liberal who got mugged”
• “Cutting edge is great until one gets cut”
• “Loss aversion is the pillar of conservatism”
• Don’t limit learning. Don’t do “for dummies.” Cognitively reframe the approach to knowing you are learning and are going to be good at it soon.
• “Great products are made by people who use them or care about them.”
• “Rails 4 will change things and break things”
• “Stay young, stay curious, stay hippie. Don’t fight change or progress”

Using Backbone.js – Sarah Mei @sarahmei

• Frontend frameworks (ember, backbone, handlebars) promote convention.
• Don’t do a full rewrite. Incrementally rewrite.
bit.ly/JQjXCX

Machine Learning – Andrew Cantino @tectonic, Ryan Stout @ryanstout

• Methods: Decision Tree, Support Vector Machine (SVM). Naïve Bays, Neural Nets
• Curse of Dimensionality: there is an exponential relation between the number of functions one has to the amount of data required.
• Overfitting: don’t memorize, generalize. Don’t test on training data, split and cross.
• Sentiment classification: build training data, extract features, use dictionary to generate word vector, classify, evaluate, improve.
• Resources: Stanford Machine Learning class
• Tools: Weka, libsvm, liblinear, vowelpal, wabbit, recommendify
blog.andrewcantino.com/blog/2012/04/22/machine-learning-project-ideas/

Rails Engines Patterns – Andy Maleh @andymaleh

• What: gem + MVC stack elements extracting common logic
• Goal of pattern: keep engine code agnostic of app customization, prevent bidirectional dependency, prevent app specific logic.
• Solves: need for multiple apps to share basic crud for a given model, common association, common logic
• Costs: overhead in new gem, about 30 minutes, constantly switch between app/engine when developing some features, upgrade gemfile refs every commit.
• Benefits: makes tests faster, code reuse across all apps at all layers, maintainable in independent project code bases with clearly defined boundaries.
• Engines VS services: engines are better for domain dependent MVC reuse in an environment where the consumer is inside the organization, services are much more expensive and better when reusing large MVC components that depend on domain data, could potentially need to scale independently of the db, and are consumed by outside organizations.
• *Talking afterwords: Andy has moved organizations from tightly coupled legacy systems, such as a single Java app, into a decoupled rails architecture by TDDing a new system, piece-by-piece, engine-by-engine, app-by-app, using ActiveModel mapped to the legacy db. The end result was a new system with less bugs, thanks to TDD, that was easier to maintain and faster to develop new features on.
bit.ly/IJgqTL

The Future of Sass – Hampton Catlin @hcatlin

• 3.2 new: placeholder selector, mixin content/blocks for mixins, @media crazy, load paths
• Depreciated: & foo, use @extends
• Scss not Sass
• libsass (SassC), but whatevs, ruby’s got it already

Faster rails apps – Ilya Grigorik @igrigorik – Google

• Measure all things.
• Histogram all things. Averages are misleading.
• Know what to measure.
• Measure perceived latency, nothing else matters.
• Optimize from the user’s perspective.
• Mobile web is x1.5 slower
bit.ly/faster-rails

Sustainable productivity – Xavier Shay @xshay – Square

• It’s easy to start, hard to maintain.
• When code doesn’t look consistent, productivity goes down.
• Reify: make code more concrete or real.
• Class documentation – describe the bigger picture.
• Questions make me less productive.
• Smart people mess up.
• Error rates multiply.
• Be on your guard, even when fixing critical bugs.
• Don’t make the code more complex.
• Anything that can be checked by machine, it should be. Catch the rest with pairing and code review. Tool: cane.
• Tools aren’t perfect. Conform anyway. There is tremendous value in 0 errors from a tool.
• Do not depend on code you cannot change. Shim it. Define the relationship in your terms.
• Group and access aggregates via root. Bi-directional associations are a symptom of ignoring aggregate roots.
• Reading: Domain Driven Design, Clean Code, A Guide to Testing Rails Applications, Growing OO Software, Guided by Tests, destroyallsoftware.com
• Tests Redefined: Integration, Functional, Unit. Unit tests do not touch the db, network or file system. VS Acceptance, Integration, Unit. Does the whole system work, does code work against code we can’t change, do methods do the right thing/are they convenient?
• *Talking afterwords: The hardest part of dealing with legacy systems is figuring out how to split them up into manageable, replaceable chunks. Put shims in between the old and new. Aim for a service/rails engine oriented architecture. Hide the ugly. Identify and utilize a single data source for everything. Use ActiveRelation to get data from anywhere.
youtube.com/watch?v=84ewfGEojsw

Zero Downtime Deploy – Pedro Belo @ped

• No MySQL + isam. Doesn’t work as rails expects.
• InnoDB is O(n) on add/remove/update columns and add/remove indexes.
• Postgres for the win: O(n) on column updates, constant on everything else if careful.
• github.com/freels/table_migrator for data, but requires acts_as_paranoid and is too old, so just use it as a reference.
• At least 2 processes, watch capacity.
• Cycle the webserver gracefully.
pedro.herokuapp.com/past/2012/4/24/zero_downtime_deploys_for_rails_apps_slides

Rails: The next 5 years – Yehuda Katz @wycats

• Conventions are good.
• Conventions are part of a bigger solution to help the developer.
• Conventions hide solved problems when working on another problem.
• Conventions help developers develop similar looking code.
• Good architecture enables future features.
• Features Now + Features Later ÷ (Cost Now + Maintenance)
• Any convention is better than no convention.
• Trivial choices are the enemy.
• Active Model Serializers standardize/bring convention to JSON APIs.
• A note on node: insert Paul Graham quote – the web lets us pick the best tool/language for the job. Is javascript the best server side language? Just b/c it won on the client, doesn’t mean it should win on the server.
• Rails developers are interested in working on startupy apps, not enterprise apps. What enables this?
dl.dropbox.com/u/2285145/The%20Next%20Five%20Years.pdf

faster rails boot ruby 1.9.3 + debugger

run this to install rbenv and ruby 1.9.3:

curl -s https://raw.github.com/gist/2369146/rbenvosx10.7.sh | sh

or by hand, with rbenv and ruby-build:

curl https://raw.github.com/gist/1688857/rbenv.sh | sh && rbenv global 1.9.3-p125-perf

or with rvm:

rvm get head && rvm reinstall 1.9.3-perf --patch falcon,debug --force-autoconf -j 3

finally (from the docs):
Putting the following in your shell config (eg. `~/.bash_profile`) will make Rails even faster, but will increase its memory footprint:

echo "# https://raw.github.com/gist/1688857
export RUBY_HEAP_MIN_SLOTS=1000000
export RUBY_HEAP_SLOTS_INCREMENT=1000000
export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
export RUBY_GC_MALLOC_LIMIT=1000000000
export RUBY_HEAP_FREE_MIN=500000" >> ~/.bash_profile

mad props to fortnight labs for pointing this out

full documentation here:

https://raw.github.com/gist/2369146/rbenvosx10.7.sh

https://raw.github.com/gist/1688857

homebrew — fix that nasty mysql.server start segfault

after install xcode 4.3 mysql.server start was shouting back at me and refusing to start:
ERROR! The server quit without updating PID file

installing the old, non llvm, gcc binaries and re-installing seemed calm the beast


brew remove mysql
brew cleanup
brew update
brew upgrade
brew install https://raw.github.com/Homebrew/homebrew-dupes/master/apple-gcc42.rb
brew install mysql

also, dont forget to follow the instructions after install


unset TMPDIR
mysql_install_db --verbose --user=`whoami` --basedir="$(brew --prefix mysql)" --datadir=/usr/local/var/mysql --tmpdir=/tmp

read/write filesystem on rooted android for an ad-free android experience

Picture this. You’ve got your hot new rooted android phone and you want to mess with the OS. Say, replace the hosts file with something like this, to block all those pesky ads that pop up so inopportunely in every app you’ve ever installed.

Ok, you’ve logged into to your phone with adb shell or run su - from the terminal emulator.

The downloaded hosts file has just finished after you ran something like this:

cd /mnt/sdcard/download && wget http://someonewhocares.org/hosts/zero/hosts

You’ve backed up your hosts file (like this cp /etc/hosts /mnt/sdcard/hosts-original) in case something goes wrong you can just cp /mnt/sdcard/hosts-original /etc/hosts to restore your original host file.

Now you’re ready to install your freshly downloaded super ad-blocking epic and awesome new host file.

You run cp /mnt/sdcard/download/hosts/etc/hosts only to find you’ve got a read only file system on your hands.

Have no fear, remount is here.

First, lets figure out what we need to remount, running ls -lah tells us where /etc is symlinked to:

drwxrwx--x 1 system system 2.0K Jan 26 09:44 data
-rw-r--r-- 1 root root 118 Dec 31 1969 default.prop
drwxr-xr-x 12 root root 1.3K Feb 14 09:27 dev
lrwxrwxrwx 1 root root 11 Feb 14 09:27 etc -> /system/etc
-rwxr-x--- 1 root root 92.2K Dec 31 1969 init
-rwxr-x--- 1 root root 1.6K Dec 31 1969 init.goldfish.rc

Ok. We need to remount /system, lets figure out where that is by running cat /proc/mounts | grep system

/dev/block/mtdblock4 /system yaffs2 ro,relatime 0 0

Now that we know /system is mounting /dev/block/mtdblock4, we can run the magical remount command:

mount -o rw,remount -t yaffs2 /dev/block/mtdblock4

we just need to run the copy once more:

cp /mnt/sdcard/download/hosts.txt /etc/hosts

and voila! your android is now ad-free (mostly)

comment with questions or a how to for iPhone

Turn Wikipedia Back On

Yep SOPA and PIPA are bad, but life without Wikipedia? I just can’t handle that.

Here is how you fix it:

1. Create a bookmark and name it “Unhide Wikipedia”

2. Copy and paste this into the url:

javascript:(function()%7B$('#mw-sopaOverlay').hide();$('#content, #mw-head, #mw-panel, #footer').show();%7D)()

It might look something like this:

3. Visit a Wikipedia page and click your new bookmarklet.

Finally, don’t forget to write somebody and tell them not to destroy the internet.

compiling qt4 on centos 5

when the favor of the linux gods shine upon you, all that is needed to build code requiring qt is:

sudo yum install qt4 qt4-devel

for some reason though when building pyqt 4.9 with this setup, the build works, however QNetworkProxy and various other classes are missing.

so, began the quest to build qt4 and pyqt 4.9 from source

long story short you need to remove -Werror from all the makefiles before building


wget http://get.qt.nokia.com/qt/source/qt-everywhere-opensource-src-4.8.0.tar.gz
tar -xzf qt-everywhere-opensource-src-4.8.0.tar.gz
find . | xargs grep 'Werror' -sl |grep Makefile | xargs sed -i 's/-Werror//g'
./configure
make
sudo make install

then you can build pyqt with your new qt install

wget http://www.riverbankcomputing.co.uk/static/Downloads/PyQt4/PyQt-x11-gpl-4.9.tar.gz
tar -xzf PyQt-x11-gpl-4.9.tar.gz
cd PyQt-x11-gpl-4.9
python2.7 configure.py -q /usr/local/Trolltech/Qt-4.8.0/bin/qmake #or instead of python2.7 use the python for which you wish to build pyqt
make
sudo make install

if weird things happen after both build succesfully, make clean and try again

pip install pymongo with xcode 4

xcode 4 kills PPC support. in other words, xcode 4 breaks pip install [stuff] if stuff tries to compile for PPC

do this to fix it:
env ARCHFLAGS=’-arch i386 -arch x86_64′ [pip install command]

e.g.

env ARCHFLAGS='-arch i386 -arch x86_64' pip install pymongo

or e.g.

env ARCHFLAGS='-arch i386 -arch x86_64' pip install http://pypi.python.org/packages/source/n/nltk/nltk-2.0.1rc1.tar.gz

interactive python history — better titled “gimme some up arrow love”

I have a mac. My mac’s python shell doesn’t bring up the last command when I press the up arrow.

Yeah, yeah I know. ctrl-p. but seriously, I’ve lived my whole life with the up arrow. I can’t just go ctrl-p’in all my last commands. Though my brain may benefit from this exercise, my temperament will not.

Here is the fix:

sudo easy_install -a readline

Then setup your PYTHONSTARTUP env var to point somewhere, like ~/.pythonrc

e.g.

export PYTHONSTARTUP=$HOME/.pythonrc

and put something like this in it:

import atexit
import os
import re
import readline
import rlcompleter
import socket
import _socket
import sys
import time
import timeit

history = os.path.expanduser(‘~/.python_history’)
with file(history, ‘a’):
os.utime(history, None)
readline.read_history_file(history)
readline.parse_and_bind(‘tab: complete’)
atexit.register(readline.write_history_file, history)

def t(*args):
return timeit.Timer(*args).timeit()

Leave me some comments if you find problems or have suggestions. I read em all.

I didn’t figure this out, just perfected it. Massive win:

http://valueerror.wordpress.com/2009/11/03/python-shell-history-autocompletion-and-rc-file/

http://stackoverflow.com/questions/893053/python-shell-arrow-keys-do-not-work-on-remote-machine

http://stackoverflow.com/questions/4289937/how-to-repeat-last-command-in-python-interpreter-shell

Centos 5.5 yum repo sha256 fix

New yum repos with sha256 hashing break yum+python2.4, which can’t support them.

Installing hudson from their new repo from example, will yield this:

http://hudson-ci.org/redhat/repodata/primary.xml.gz: [Errno -3] Error performing checksum
Trying other mirror.
Error: failure: repodata/primary.xml.gz from hudson: [Errno 256] No more mirrors to try.

Fortunately, we can make python2.4 play nice by installing the python library hashlib.

If I didn’t already have easy_install and pip pointing to python26 you could probably:
easy_install hashlib
or
pip install hashlib

but its easy enough to install it from source, just do this:

sudo yum install python-devel
wget http://pypi.python.org/packages/source/h/hashlib/hashlib-20081119.zip#md5=46e59db6d5cac34b3344ef0c30d0897f
unzip hashlib-20081119.zip
cd hashlib-20081119
python setup.py install

Copyright © Nathan 0xCE5A6D8C