Database Cleaning with Cucumber, Machinist 2, Rails 3, DatabaseCleaner, Mysql2 and Mongoid

This combo does not work out of the box right now. At a minimum, in order to use mysql 2, you’ll have to get this commit, which currently involves using the git source for db cleaner in your Gemfile.

You’ll also want to comment out the db cleaner stuff in env.rb and switch it to support/database_cleaner.rb and do it like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# DB Cleaner currently being used for AR
Mongoid.master.collections.select do |collection|
  collection.name !~ /system/
end.each(&:drop)

DatabaseCleaner.strategy = :truncation

Before do
  DatabaseCleaner.start
end

After do
  DatabaseCleaner.clean
end

There may be a better way to do this, but I’m not aware of it.

This solves issues like:

1
2
3
4
5
6
7

deadlock 0x1001c1258: sleep:J(0x1072ed8f0) (main) - /Users/foo/.rvm/gems/ree-1.8.7-
2010.02/gems/machinist-2.0.0.beta2/lib/machinist/active_record/blueprint.rb:33
INTERNAL ERROR!!! Thread( ): deadlock
        /Users/foo/.rvm/gems/ree-1.8.7-2010.02/gems/machinist-
2.0.0.beta2/lib/machinist/active_record/blueprint.rb:33:in `value'

and

1
2
3
4

NoMethodError Exception: undefined method `truncate_table' for #
<ActiveRecord::ConnectionAdapters::Mysql2Adapter:>

August 23, 2010

Questions to ask oneself

Most people do not ask themselves these questions constantly. Imagine if everyone did!

a) What is my goal?

b) What am I doing right now to move my project toward that goal?

c) Am I waiting on anyone else? Have I told them that?

d) Is anyone else waiting on me?

e) Is it clear to everyone what to do next?

f) Is it clear to everyone what we are working on?

g) Have I told everyone when I expect to finish?

h) Has everyone told me when they expect to finish?

i) Has anything about the project changed on my side? If so, have I told everyone else?

August 11, 2010

Recursively grep files for a string and show line numbers

1
2
3

grep -nir "foo" .

August 06, 2010

Sending mail from the command line in linux

When testing things like DKIM and and spf records it’s useful to be able to send mail from the command line.

1
2
3
4
5
6
7

root@server:~# sendmail someone@gmail.com
FROM: you@yourserver.com
SUBJECT: hi
the body
.

Alternatively, it’s faster to do make a file called message.txt with something like:

1
2
3
4
5
6
7

to:someone@gmail.com
from:you@yourserver.com
subject:DKIM

This is my message.

And then

1
2
3

cat message.txt | sendmail -t

That way you can send it over and over again without having to spend any time.

August 05, 2010

Mac Vim Speed Tips

Today I spent a few minutes making things better for myself with MacVim.

First, I upgraded to snapshot 52 of MacVim. There hasn’t been a stable release of MacVim since 2008, but development is ongoing. There are about 2 years of MacVim development that you can take advantage of if you use the snapshots.

Snapshot 52 was released in March 2010 makes rendering a lot faster, and also includes an even faster experimental rendering mode, but this did anti-aliasing which I didn’t like so I didn’t use it.

If you use the stable version of MacVim, then you are no doubt aware how slow the rendering is. Highly worth it to upgrade!

Second, I got the PeepOpen application and added:

1
2
3
4
5
6
7

"""""""""""""""""""""""""""""""""""""""""""""""""
" PeepOpen                                      "
" Project search                               "
"""""""""""""""""""""""""""""""""""""""""""""""""
map <leader>t :PeepOpen<CR>

to my .vimrc

These two changes make opening files about 100x faster due to faster rendering and faster search. Fuzzyfinder textmate was bugging me due to issues with 1.9.2 I think, as well as a extreme slowness in a my rather large application.

August 05, 2010

Rails 3 Cucumber / Machinist Tutorial: Machinist with Cucumber in 10 minutes

Erwin was kind enough to update my cucumber and machinist tutorial for Rails 3. This should be really useful for people who want to use cucumber to test with Rails 3. He notes that this is current with Rails3.rc and ruby 1.9.2.rc.1.

You can find his full repo here.

Rails 3 cucumber tutorial

So, here’s how to start using machinist with cucumber in a rails 3 app. I’ll use a simple app that deals with strawberries.

Start a rails app

1
2
3
4
5
6
7
8
9
10
11
12
13

# sudo gem install rails --pre --source=http://gems.rubyonrails.org
# or use rvm to test Rails 3

rails new strawberries
cd strawberries
#  install http://github.com/ryanb/nifty-generators
#To use Nifty Generators with Rails 3 you will need to include it in your Gemfile.
  gem "nifty-generators"

rails g nifty:layout
# other generators :  nifty:scaffold, nifty:config, nifty:authentification

Add them to your Gemfile, so that other folks on your team can easily get them.

1
2
3
4
5
6
7
8
9
10
11

gem 'database_cleaner'
gem 'cucumber-rails'
gem 'cucumber'
gem 'rspec-rails'
gem 'spork'
gem 'launchy   # so we can use : Then show me the page

gem 'machinist', '>= 2.0.0.beta1'
gem 'faker'

1
2
3
4
5
6
7
8

#Then install the gems by running:
 bundle install

# generate schema
rake db:migrate
# or alter /config/boot.rb if you do not intend to use a database

Generate the files

From your rails app’s root, enter:

1
2
3
4
5
6
7
8
9


rails g cucumber:install --capybara
rails g machinist:install

touch features/support/blueprints.rb
# add this line in it
require 'machinist/active_record'

Write a story

Now that we have our environment setup, let’s add the stories. These are just sketches of how our app should behave.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

@wip : cucumber tag for work-in-progress


# features/strawberries.feature
Feature: Live
  In order to live
  people
  want to be able to eat strawberries
 
@wip 
  Scenario: Eating strawberries
    Given a strawberry that is "blue"
    When I go to the homepage
    Then I should see "There is 1 strawberry"
    When I follow "eat the blue strawberry"
    Then I should see "Strawberry eaten!"

Implement steps

It’s a good idea to implement steps first, so that you can have failing steps to work with. Cucumber will provide step definitions for us, so just enter:

1
2
3

rake cucumber:wip

Now we can add the step definition for the strawberries.

1
2
3
4
5
6
7
8


#features/step_definitions/strawberry_steps.rb
Given /^a strawberry that is "(.*)"$/ do |color|
  Strawberry.make!(:color => color)
end
  

Now let’s try rake cucumber:wip This time we get

1
2
3
4

1
uninitialized constant Strawberry (NameError)

Which means it’s time to start adding code.

Implement feature and add blueprints</h3

Now that we’ve written some tests, we can go ahead and build this.

1
2
3
4

rails g nifty:scaffold strawberry color:string
rake db:migrate

Again, I’m using the nifty generator, which saves lots of time for making quick demo apps. Also, we’re going to want to add a strawberry blueprint, so that we can use Machinist. So, again, let’s try

1
2
3
4

rake cucumber:wip
# => No master blueprint defined for class Strawberry

This time we get No blueprint for class Strawberry (RuntimeError). So, let’s add a blueprint:

1
2
3
4
5
6
7

#features/support/blueprints.rb

Strawberry.blueprint do
  color {Faker::Lorem.words(1).first.downcase}
end

This gives us the ability to say Strawberry.make, which will create a new strawberry with a random color. The nice part about machinist is that we can do both Strawberry.make and Strawberry.make(:color => ’red’), which becomes really useful when you have a model with a lot of validations that you don’t want to have to specify every time. Machinist lets you focus on specifying only what’s important to the specific test. Now, at this point, since I used the nifty_scaffold gem, we just have to change some text to make the feature pass, and we’re done.

So, now that we have a blueprint, let’s try rake:features again. We get undefined method `root_path’ for # (NoMethodError).

So, to fix this, let’s add a root path and delete public/index.html.

1
2
3
4
5

# config/routes.rb

root :to => "strawberries#index"

And then try rake features again. This time we get:

1
2
3
4
5

Then I should see "There is 1 strawberry"  # features/step_definitions/web_steps.rb:89
      expected: /There is 1 strawberry/m,
           got: "<! #etc etc

So, what’s happening here is that cucumber/webrat are going to our home page, but they aren’t finding the text that we wrote that we wanted to see in the story. So, let’s change it.

1
2
3
4

# app/views/strawberries/index.html
<h1>There is <%= "#{Strawberry.count}" -%> strawberry</h1>

Now if we try rake features again we get Could not find link with text or title or id “eat the blue strawberry” (Webrat::NotFoundError) So we add that link

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

<h1>There is <%= "#{Strawberry.count}" -%> strawberry</h1>
<table>
  <tr>
    <th>Color</th>
  </tr>
  <% @strawberries.each do |strawberry| %>
    <tr>
      <td><%=h strawberry.color %></td>
      <td><%= link_to "Show", strawberry %></td>
      <td><%= link_to "Edit", edit_strawberry_path(strawberry) %></td>
      <td><%= link_to " eat the #{strawberry.color} strawberry", strawberry, 
:confirm => 'Are you sure?', :method => :delete %></td>
    </tr>
  <% end %>
</table>

If I try rake cucumber:wip again, I see that now only the last step is failing.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

Then I should see “Strawberry eaten!” # features/step_definitions/web_steps.rb
expected: /Strawberry eaten!/m,

000

So, let’s fix that too.

---RUBY

# app/controllers/strawberries_controller.rb

  def destroy
    @strawberry = Strawberry.find(params[:id])
    @strawberry.destroy
    flash[:notice] = "Strawberry eaten!" #change the flash
    redirect_to strawberries_url
  end

So, now we have 5 passing steps

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

…./strawberries$ rake cucumber:wip

Feature: Live
In order to live  # features/strawberries.feature
  people
  want to be able to eat strawberries
  Scenario: Eating strawberries                # features/strawberries.feature:5
    Given a strawberry that is "blue"          # features/step_definitions/strawberry_steps.rb:2
    When I go to the homepage                  # features/step_definitions/webrat_steps.rb:6
    Then I should see "There is 1 strawberry"  # features/step_definitions/webrat_steps.rb:89
    When I follow "eat the blue strawberry"    # features/step_definitions/webrat_steps.rb:14
    Then I should see "Strawberry eaten!"      # features/step_definitions/webrat_steps.rb:89


1 scenario
5 steps passed

Machinist note So, hopefully this is enough for you to get started with. The value of machinist really comes later, when you have a model with more than 1 field, and when you need to make a bunch without wanting to specify everything every time. For example:

1
2
3
4
5
6
7

User.blueprint do
  password {'password'}
  password_confirmation {'password'}
  name {Faker::Lorem.words(1).first}
end

Here, it would be annoying to have to specify the passwords each time, so it’s easier to let machinist do it, so that you can just say User.make!, or User.make!(:name => ’mischa’).

August 04, 2010

Getting started with Chef tutorial

Chef is a tool for turning your servers into code. Chef is awesome. It took me a while to realize it, though.

After setting up a mysql slave server for the fifth time, I decided that I was done manually configuring servers. Chef seemed like an obvious choice. It took me about two days of hacking to figure out exactly how to get it to work.

I ended up having to use a ton of different resources to get started. Googling for what to do with Chef is non-trivial. If all of the resources had been in the same place, it probably only would have taken me about a day.

This post aims to help small companies with 5-10 servers get started with chef. I recommend following the directions in order. Before you know it, you’ll be cookin’ like a master!

For a brief overview of what Chef is for check out Opscode’s blip.tv stuff (Chef creator).

For this tutorial, we’ll be using the Opscode platform, which saves you the trouble of setting up an opscode Chef server. If you want, you can do that later.

Chef has two parts: a client and a server. Chef-client runs on all of your machines, and they reach out to the server for recipes or configurations.

So, once you’ve watched the first video register for the opscode platform here

After that, instead of going to the Chef wiki (it’s overly complex for newcomers), “check out the “getting started guide. in Opscode’s help app

If you prefer a Railscast style video, you can also watch this video which is just a walk through of the getting started guide above.

I recommend following through with the entire getting started guide. I initially wanted to just use chef-solo, which seemed to make more sense, but it turns out it doesn’t because you miss out on all kinds of cool features.

I advise against adding a bunch of cookbooks to your chef-repo early on, since there are weird dependencies and other issues that can make things more confusing than they need to be initially.

The problem with the getting started guide is that it doesn’t tell you how to actually configure a node outside of your own machine. Doing this is actually pretty easy, but somewhat undocumented.

First, get a blank server somewhere. Note that if you’re using your local machine, it didn’t work for me to just use VMWare Fusion virtual machines. The Opscode platform wouldn’t recognize them.

Instead, I recommend just creating a small server on Linode/Rackspace/Slicehost. I used Ubuntu Lucid, so I’d recommend that as well if you want to follow along. However, any Chef supported distro should work. Once the new server is imaged and you get your root password, do the following:

1
2
3
4
5
6


gem install net-ssh-multi
knife bootstrap 172.x.x.x 


The net-ssh-multi gem will save you from a confusing error.

Knife, as you learned in the getting started guide, is the Chef command line tool. Shef is to Chef as script/console is to Rails while Knife is a bit more like Capistrano, sorta.

The knife bootstrap command will try to login to the given IP as root (you’ll need the password, unless you’re using ec2 or something where keys are set). It will then setup the node (the server) to run chef-client. If you followed the getting started guide correctly, it will also upload the ssl stuff.

Once knife bootstrap is finished, login to the server using ssh.

Then run

1
2
3

chef-client

This will connect to your opscode Chef server and run any recipes that you’ve specified. At this point, you should not have specified any recipes, so it should run cleanly.

Next, it’s time to make a role. You can do this on the opscode web ui, but I personally prefer just editing raw JSON than clicking around.

So, in the roles roles folder add a new file called base.json.

Put the following json into the file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

{
    "name": "base",
    "default_attributes": {
      "chef": {
        "server_url": "https://api.opscode.com/organizations/#{YOUR COMPANY!}",
        "cache_path": "/var/chef/cache",
        "backup_path": "/var/chef/backup",
        "validation_client_name": "#{YOUR COMPANY!}-validator",
        "run_path": "/var/chef"
      }
    },
    "json_class": "Chef::Role",
    "run_list": [
      "recipe[ubuntu::default]",
      "recipe[build-essential::default]",
      "recipe[openssl::default]",
      "recipe[chef::client]",
      "recipe[chef::delete_validation]",
      "recipe[git::default]"
    ],
    "description": "Basic Server",
    "chef_type": "role",
    "override_attributes": {
    }
  }

Be sure to replace the stuff in #{} with your info. Then run:

1
2
3

knife role from file roles/db-slave.json 

This will upload the role to the chef server.

Now we need to download those cookbooks. For each of the items in the run_list array above, do

1
2
3

knife cookbook site vendor #{name}

For example

1
2
3

knife cookbook site vendor ubuntu

This will download the cookbooks to your chef-repo and create a git commit.

Next, it’s time to upload your cookbooks:

1
2
3
4
5
6
7
8

knife cookbooks upload -a    # uploads chef cookbooks with knife

# OR

rake upload_cookbooks


Both of the above commands will upload your cookbooks.

At this point, it’s worthwhile to start getting really familiar with the output of knife when you don’t enter any arguments. It lists all the commands, which you will have to learn just like you would rails commands. It takes a while, but they start to feel natural.

Next, let’s try to run chef-client on the node. As root, on your new node, run:

1
2
3

chef-client

It should run cleanly. If it doesn’t feel free to e-mail me at f.mischa@gmail.com and I will update this post.

If it does run cleanly: Congratulations! You’re cooking with Chef. Ohai!

From here on, the workflow for me is basically just:

[workstation] knife cookbook site vendor #{cookbook} -d # The -d flag will install dependencies [workstation] knife cookbook upload #{cookbook} [workstation] add the recipie from the cookbook that you want to use to the base role [workstation] update the base role with the command above (from file) [node] run chef-client [node] check out what errors you get [workstation] try to fix them

For many recipies, especially the 37signals ones here there is basically no documentation. This means that you will often have to read through the entire cookbook to figure out how you need to configure it. It’s very much trial and error. The best strategy is to read through the cookbook, creating a json hash for w/e data you’ll need.

The 37signals users cookbook is really great, so I will quickly run through how to set it up, as an example of how to figure out undocumented recipes.

First, grab this gist. and put in your chef repo in chef-repo/tasks and add the rake file stuff to your chef-repo’s rakefile.

That gist will help you deal with databags sanely.

For the 37signals users recipe, clone the 37s cookbooks repo and then copy the users cookbook to your cookbooks directory in your chef-repo.

Next, in chef-repo/databags, you’ll need two files.

databags/groups.json:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

[
  {
    "id": "admin"
  },
  {
    "id": "other"
  },
  {
    "id":"whatever"
  }

]

You can start with just admin for now.

Also, you’ll need databags/users.json:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

[
  {
    "id": "mischa",
    "groups": [
      "admin"
    ],
    "uid": 7777,
    "local_files": true,
    "full_name": "Mischa",
    "shell": "/bin/bash",
    "ssh_key": "my key",
    "password": "my pw"
  }
]


Change this to your info, obviously. For password, you can get the value to put in by doing:

1
2
3

openssl passwd -1

Note that that’s a one, not an l. That will give you a value to put there. For the ssh key, use your ~/.ssh/id_rsa.pub If you dont have one of those, not sure why you’re reading this, but go over to github and they will show you how to make one.

Now it’s time to use our handy rake task from the Gist. Note that it expects a ~/.chef directory, so you may want to link or copy your .chef stuff from your chef repo to your home directory.

1
2
3

rake load_data_bags

This will upload our users and groups data bags to the chef server. Now you just have to:

1
2
3

knife cookbook upload users

Finally, update your roles/base.json to:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

{
    "name": "base",
    "default_attributes": {
      "active_groups": [
        "admin"
      ]
      },
      "chef": {
        "server_url": "https://api.opscode.com/organizations/foo",
        "cache_path": "/var/chef/cache",
        "backup_path": "/var/chef/backup",
        "validation_client_name": "foo-validator",
        "run_path": "/var/chef"
      }
    },
    "json_class": "Chef::Role",
    "run_list": [
      "recipe[ubuntu::default]",
      "recipe[build-essential::default]",
      "recipe[openssl::default]",
      "recipe[chef::client]",
      "recipe[chef::delete_validation]",
      "recipe[git::default]",
      "recipe[users::default]"
    ],
    "description": "Basic Server",
    "chef_type": "role",
    "override_attributes": {
    }
  }

And then upload role the using knife role from file as above.

And then run chef-client on the node. This should install users and setup your ssh stuff. Sometimes chef-client fails on the first run because it can’t fine some package. Running it again somehow fixes this, and it can find the package.

Next steps

From here, the world is yours. You should look through all the opscode cookbooks and the 37s ones and see which you like. I recommend starting small, since it can get pretty complicated fast.

My entire base configuration is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

{
    "name": "base",
    "default_attributes": {
      "active_groups": [
        "admin"
      ],
      "active_sudo_groups": {
        "admin": {
          "enabled": true
        }
      },
      "chef": {
        "server_url": "https://api.opscode.com/organizations/foo",
        "cache_path": "/var/chef/cache",
        "backup_path": "/var/chef/backup",
        "validation_client_name": "foo-validator",
        "run_path": "/var/chef"
      }
    },
    "json_class": "Chef::Role",
    "run_list": [
      "recipe[ubuntu::default]",
      "recipe[build-essential::default]",
      "recipe[openssl::default]",
      "recipe[chef::client]",
      "recipe[chef::delete_validation]",
      "recipe[git::default]",
      "recipe[sysadmin::default]",
      "recipe[munin::client]",
      "recipe[screen::default]",
      "recipe[ssh::server]",
      "recipe[iptables::default]",
      "recipe[users::default]",
      "recipe[sudo::default]"
    ],
    "description": "Basic Server",
    "chef_type": "role",
    "override_attributes": {
    }
  }

To get all of these to run, there are a bunch of dependencies:

1
2
3
4
5
6
7
8
9


chef-repo$ ls cookbooks/
        build-essential/ hosts/           mysql/           runit/           sysadmin/
37s_mysql/       chef/            iptables/        openssl/         screen/          ubuntu/
README           couchdb/         java/            perl/            ssh/             users/
apache2/         erlang/          maatkit/         rabbitmq_chef/   ssh_known_hosts/ xml/
apt/             git/             munin/           ruby-shadow/     sudo/            zlib/

As you can see, I’m using some opscode coobooks and some 37signals ones.

Many of the recipes above required pretty heavy customization to get to work. Note that the iptables chef recipe will lock you out of your server unless you configure it properly before running it. Furthermore, the ssh recipe also required a good amount of customization. If you need help figuring out how to configure the iptables recipe, e-mail me. Unless I’m getting slammed with e-mails, I’ll be happy to help.

However, I am still pretty new at all this, so I really recommend the #chef irc channel on freenode. One of the guys there seemed pretty helpful.

For reading cookbooks, I recommend starting with the Ruby files in the recipes folder, and then looking at the templates and attributes to figure out what data you’ll need to set in the role or node.

For the 37s cookbooks, I recommend treading carefully. Some of them are for HARD CORE servers. The 37s mysql one, for example, does way more than most companies will ever need. Furthermore, they often assume things about 37s systems that will not be true for your systems most likely. For example, I think that they assume a 37s image that has some packages and stuff pre-installed or waiting in various locations.

Overall, I recommend customizing the cookbooks to suit your size and organization. I hope that this was useful, and that it saves others the pain of having to aggregate all of the getting started information. For further help, I recommend the opscode wiki and the #chef channel on freenode.

Please e-mail me if there are any problems with this tutorial. You should also subscribe and follow me on github!

Also please come work with me if you are smart and in Chicago!

July 31, 2010

Learning how to run a startup from Clear's horrible customer service

My internet stopped working today. I called clear and spent about 20 minutes trying to get them to fix it and refund me for the month. They told me that they won’t refund me unless I call them again when it starts working, and that it will take 2-24 hours to fix.

Given that I work from home usually, this really sucks. Clear also appears to have a history of bad service.

It’s inspired me to be way better with my own company.

Apologize

Clear never apologized to me. They never admitted any kind of fault on their end.

At my company, we always clearly apologize when something is our fault.

Explain what the problem is

Clear never explained to me what was actually happening and why it was broken.

My company can do a much better job is making it obvious to our customers what went wrong.

Don’t rely on customers for your issue tracking

In order to get an engineer out to fix the tower, Clear had to fill out a form that listed all the customers having trouble. The hilarious part is that THEY RELY ON CUSTOMER SUPPORT CALLS TO DETERMINE WHETHER AN AREA IS HAVING TROUBLE. I.e. if they get enough calls, then they decide to send out an engineer.

This is so silly. Why not have each modem check in every 5 minutes, and dynamically send out people to fix towers based on what modems don’t check in? I can’t imagine that would take more than a week or so to implement.

Either way, relying on your customers to tell you when you are failing is the worst strategy.

At my company, we have automated error tracking as well as other systems that constantly check to make sure things are working and alert is (munin, hoptoad, cron e-mails, lots of custom stuff).

Try to make everyone happy

Pissing off lots of customers is really costly. Word spreads.

My company can do better with this. There are occasionally cases where people are not happy with us after dealing with customer support.

Try to refund people for the opportunity cost

If your service costs $50/month, and it being broken costs a customer $100 (e.g. if they charge $100/hr), then refunding them for the 1 day that it doesn’t work makes no sense.

You have cost your customer $100, but you are only refunding them $1.60. Obviously, it would be silly to refund people based on their salary. The only thing that makes any sense then, is simply to refund people for an entire month whenever it breaks for more than an hour or so.

At my company, we generally already do this when someone has a problem. We should probably be making a greater effort to ensure that this always happens though.

If you are constantly having to argue with your customers, Something is wrong!

Clear obviously constantly has similar arguments with their customers. This indicates that something is very wrong.

It’s easy to start seeing customers as the vague masses beyond the castle walls—as people to be argued with and kept out. This is the wrong strategy though. In the long run, it’s better to deal with the underlying issues and welcome your customers in.

Solve common issues within 15 minutes to an hour

Waiting 6-24 hours for a critical service is unacceptable.

Don’t fall back on the industry standard

When I was complaining to the Clear person, she compared it to ATT and her cable company and how she has problems with them too. Are you kidding me?

Exceed expectations

Apple once replaced my phone despite it being about two weeks out of warranty. This made me really happy for like a week, and I told everyone about how awesome it was. Now I’m telling everyone about how crappy Clear is.

Don’t be sales rep driven

Clear is driven by sales, it seems. This screws the entire company, imo, because it means that in order to sell they have to have really good sales people, rather than a really great product that gets good reviews.

Make it easy to cancel

Clear is really hard to cancel. They try to lock you in as much as possible.

This likely just makes it even harder for the top management to see that their service sucks.

The easier it is to cancel, the quicker you will know about product issues.

Conclusion

I spend a lot of time trying to learn from really great companies. Learning from bad companies is great too.

July 26, 2010