Last month Brice Figureau released a little Nginx module which tracks the progress of uploads going through Nginx. Like Lighttpd's mod_uploadprogress it's very nice to have the proxy take responsibility for reporting the progress.

Read the rest of this entry

Summary

• Use STRAIGHT_JOIN if MySQL is doing something silly
• MySQL Datetimes are not slow
• MyISAM is much, much faster than InnoDB
• Consider MySQL partitioning; it's not hard
• Watch out for background optimisation and query caching
• Log everything you do, and why

The reporting application I'm working on does a date-range select against a table with ten million rows. It took over two minutes to execute. I cut it down to around two seconds without significantly changing the application. Here's how.

Read the rest of this entry

In one of our current projects, it is required that certain pages show different articles depending on the profile and network of contacts of the logged-in user.

In order to have a working prototype as soon -and Agile- as possible, we defined the logic for those viewing rules directly into our ActiveRecord models..

def article_list( user )
  articles = []
  if user.is_a? Admin #Admin can see everything
    articles += find(:all)
  # public articles, written by the user or his contacts
  elsif user.is_a? User 
    articles += find(:all, :conditions=>{:public=>true})
    articles += find(:all, :conditions=>{:author=>user})
    articles += find(:all, 
       :conditions=>["author_id IN (?)", user.friends.collect(&:id)])
  else # anonymous user can only see public articles
    articles += find(:all, :conditions=>{:public=>true})
  end
 
  articles.uniq.sort_by(&:published_at)
end

It's a prototype. No one said it had to scale.

Even though we could optimize the method and reduce the amount of hits to the database, the real risk with this approach lies in that we're trying to declare our business logic (who sees what) and our data storage strategy in the same place. One block of code attempting to deal with two separate concerns at once. Should our viewing rules change, we would have to modify our conditional logic both in the if / else structures and in the arguments we pass to ActiveRecord finders. If, on the other hand, our app becomes a success and we decide to resort to Memcached for quick data access, we must change our data storage strategy with extreme caution to avoid breaking the conditions that support our rules.

Expressing our rules in plain SQl is not a sensible option, as it probably would make the code even more intricate.

After a coffee (or ten) and a bit of fresh air things are clear. Viewing and privacy rules for the app's articles might change, or new user profiles can be added to the mix. The core concept of the project is sustained by those rules. The how and where we store those data is a detail of implementation that has little to do with the former.

We should be able to freely define rules and permissions in one place, and handle the heavy lifting elsewhere.

With a little refactoring and Ruby's transparency we came up with something like this:

Class RegisteredUser < User
  ...
  def visible_articles
    set_rules do |articles|
      articles.add :from=>self, :to=>:anybody, :status => :all
      articles.add :from=>:anybody, :to=>self, :status => :public
      articles.add :from=>contacts, :to=>:anybody, :status => :all
    end
  end
 ...
end

With this solution, the set_rules method can be reused in different contexts to define relevant rules. After processing a block of rules, it returns a simple array of options that we can pass to ActiveRecord finders or some other query-building utility that helps us minimize database hits. Yes, Ambition, I'm looking at you!

This simple DSL separates the definition of our business rules from data access -which we handle in a different layer of the application. As a side effect, this gives us much more readable code that is easy to maintain and test. Also, we get closer to the core domain of the project.

Where are the conditionals? With the proper use of inheritance -by class or modules- we can have every User profile declare their own set of straight-forward rules without the need of confusing conditions (you know it's a good time to re-think the architecture when those sneaky ifs and elses start creeping in all over the place).

Rails is in itself a DSL and suitable for most of web scenarios. Still, sometimes it makes sense to step back from your tools for a bit so you can get closer to the actual problem.

Getting Things Done (GTD) is a way of choosing what to do next. It is more efficient and less stressful than other systems.

Background

Last year my web development company, consisting of myself and an ever-increasing number of subcontractors, began expanding very quickly. While I stayed mostly on top of the situation, the stress was overwhelming.

I made the switch to GTD during Christmas 06.

Now that I'm working full-time as a developer, instead of as a sole trader, my task lists are much shorter. I've therefore made some simplifications. GTD can be used for large-scale project management, but that isn't covered here.

Benefits

GTD is both a workflow and an attitude. The core concept is that nothing lives in your brain, except for what you're currently working on. You must be merciless about this.

The real benefit, and the driver for me, is this: You get a stronger personal focus, as well as a stronger work focus.

When I'm at work, I'm really at work. When I'm at home, I'm really at home. This means more effective recovery, and more energy at work.

David Allen's book Getting Things Done, which is where I started with GTD, says this:

"In order to hang out with friends or take a long aimless walk and truly have nothing on your mind, you need to know what all your actionable items are, where they are, and that they can wait. And you need to be able to do that in a few seconds, not days."

Selecting Tasks

I have everything that I ever need to do in one place. The everything bit is important.

These items are described in the form of "next actions", or "the very next thing I can do to move this forward". They are very specific. For example, I don't have "deal with tax return"; I have "phone ATO re: missing BAS for FY06-07, 13 28 66, 9-6pm Adelaide time".

Because of this precision, I can allocate tasks to "Contexts", which are sets of prerequisites. For example, I can't perform the "tidy room" task unless I'm at home. I can't call the ATO unless it's Really Early (because they're in Australia).

This means that tasks are invisible to me unless I can perform them right now.

The workflow is:

1. Complete current task.
2. Select context.
3. Select a new task from tasks in this context. There's no advance scheduling and prioritising; in the midst of constantly shifting requirements, I trust myself to make the selection just-in-time.

Adding Tasks

That's how you work with tasks in the system. What about when a new task appears?

First, you stash it in one of your inboxes. You have the smallest number of these that you can get away with.

Your brain is absolutely not allowed to be an inbox. If you're filing things in your brain for even a second, you need another inbox.

I use two: my email account, and a notepad application on my phone, which is always with me.

Stashing an item in an inbox is very quick, and doesn't interrupt what you're doing. Whenever you have time - eg whenever you finish a task - "process" your inbox. For each item in it, you have four options:

Do it

If it's going to take less than two minutes, do it. Do it right now. Then forget about it.

The "two minutes" is a varying threshold. Anywhere between two and five works for me. I'll initiate system backups, answer quick emails, fill in my timesheets, and occasionally even make phone calls. If I spend ten minutes and kill off five two-minute tasks, I feel _good_.

I've taken this a step further, and will even break inbox queuing to do this: if a new task appears I'll sometimes stop what I'm doing to kill it off.

Delete it

GTD requires you to be realistic. Some things you'd like to do, you might have long-term plans to do, but realistically, you're never going to do them.

Don't feel bad about them. Don't lie to yourself, or to other people. Instead, say "I'm never going to do this", delete it from your inbox, and forget that it ever existed.

You can have a "Someday" context for items that will live in your brain if you don't put them somewhere else. I review mine periodically to weed out things that just aren't ever gonna happen.

Delegate it

If you give the job to someone else, the chances are you're still responsible for it. You need to add it to a "Waiting" context, with a description like "Waiting for ATO to send new form". Periodically scan this context, and see if any of these things:

• has happened, so you can remove it, or
• hasn't happened, and you need to follow it up.

If you don't put it in the "Waiting" context, it'll still live in your head. You don't want that.

Defer it

This is your option of last resort. It means "I can't do this now, but I do have to do it, and I can't give it to anyone else". Here's where you work out the very next action that you need to take in order to move this forward. Do take the time to calculate the next action; often, it's very small ("sort out my tax" is a huge job, but "email the ATO asking for a form 12" is not), and you will be able to do it now, after all.

Once you've got a next action, choose a context for it. Store it under that context in whatever system you're using.

You are now committed to doing this task.

Equipment

The Book tells you to buy a lot of things. I didn't have to.

You must, must have a small notebook that you carry with you all the time, particularly at first when you are dumping the contents of your brain. I use the built-in Notes application on my Symbian phone.

Until recently, I used Actiontastic, relying on the fact that I always have my laptop with me. I'm now in the process of migrating to the Notes application on my phone, because I'm using the laptop much less at home.

I didn't get hung up on syncing. I always have my phone. Why would I need to sync it to anything else?

I have all my paperwork in a single box - yes, I use shoebox filing. This works because everything I care about is also in iPhoto. I use a digital camera rather than a scanner, because it's orders of magnitude faster and the quality is "good enough". iPhoto is crap for this, but it's "good enough" - I use the description field to insert keywords.

I have a single "reference" folder in my email account. I find that gmail's search is good enough that I don't even need to use labels. Anything that I'm not sure I'll need, I delete.

The Book covers some other areas, such as the "hard landscape" - tasks that must occur at certain times - and "tickler files" (tasks you'd like to forget about for a while, but do need to revisit). I use my phone's calendar for both. I sync with my partner, friends, and co-workers using the low-tech but highly-effective "what have we got on Tuesday?" method.

These are the tools you do need:

Getting Things Done, by David Allen. It's an inspiring read, and packed full of details that are worth knowing about even if you don't use them. GTD is a style, not a religion.

Actiontastic, if you're going to keep your GTD lists on your Mac. The Omni Group is working on something called OmniFocus, but it's still in early beta.

As far as I know there are no GTD tools, or even suitable Todo-list tools, for Symbian phones. If there's enough demand, I'll write one - let me know ('gwyn' at this domain).

Making The Switch

One of the guys here at New Bamboo had previously tried to switch to GTD, but bailed out when The Book told him to take two days off to sort out his life. You don't have to do that.

But you should.

I read The Book and made the switch in the Christmas-New Year break last year. The timing worked well for me, and allowed me to go through everything, getting the full clear-head benefit. Yes, I had tasks like "charge my phone" and "tidy my desk". It's like tidying up; once you start, you go a little crazy.

Carrying a notebook is critical. I found that once I started the dump, I couldn't stop it.

Don't get sidetracked into software. Half of New Bamboo reacted to my initial presentation by planning to write software to track their tasks. This defeats the purpose, which is to increase productivity. This isn't about software, or three-colour stationery and ring binders. This is about habits.

Summary

To summarize:

• Get everything in one place.
• Think of your tasks in terms of the Next Actions that are needed to complete them.
• Classify your tasks into Contexts.

GTD takes some effort to maintain, but you'll feel it when you start to slip: the stress will come back.

Postscript: GTD at New Bamboo

The material above is drawn from an internal presentation ("GTD: Task Management For Ninjas") that I gave for New Bamboo's Talk At Two program. A month later, I did a quick survey to see if it stuck. While there's only one new full-blown implementation, nearly everybody found something useful to incorporate into their existing systems.

Three people didn't attempt GTD. Two of those weren't here for the presentation.

Damien tried it, but found that my stripped-down version didn't track larger projects. He's using the "errands" context, though, with a separate list that he looks at on his way out the door.

Pablo's using a series of paper lists, and is enjoying lower stress and the sense of achievement that comes from crossing things out. He's also been inspired to collect old tasks and to evaluate them realistically.

Ismael's using GTD's aggressive single-task focus, but finds he can store his lists in his brain without stressing himself out.

Max is running a full implementation, using Actiontastic, but he's still using his brain as an inbox for when he's not near a computer.

Matt started using GTD, then found he no longer needed it.

He stopped because he Got Everything Done.