Wednesday, December 15, 2010

VIM Command Line History Window

Ever been recording a macro when all of a sudden an unexpected window pops up out of nowhere in the bottom of your window, the contents of which appear to resemble the last few commands you've typed?

Well, it's definitely happened to me. This isn't exactly the sort of thing for which you can search online -- what sort of query can succinctly describe that? Well, today as part of another quest, I finally discovered what's going on!

VIM supports command line history (as can be most easily observed by pressing up or down in command line mode, or just typing ":history"). What I didn't know was that when in Command Mode, typing "q:" will open your command history in a sub-window. You can then navigate this window, make changes (in Normal mode), and re-execute the original commands or (and this is the cool part) modified commands simply by pressing enter! (from Command Mode).

To exit out of the sub-window, just CTRL+C (twice) as you would to cancel out of Command Line mode.

Monday, November 15, 2010

Ignore Missing Files with tar

I've been experiencing an annoying problem with tar in which it is not creating archives if at least one file in the tar list does not exist. This behavior is totally acceptable and appreciated when I am tarring on demand. However, in this case, I am running a command across several machines each of which will have a different sample of a set of files and I need this done as a cron. And to make matters more complicated the file set involves specific files in a set of directories.
# The novice command
bash$ tar cfz yesterday-logs.tgz \
log-type1/log-type1-2011-11-15.log \
log-type2/log-type2-2011-11-15.log \
log-type3/log-type3-2011-11-15.log \
This command will work fine if all the directories exist. However, if at least one of the directories or files does not exist, I will experience the following error and an empty tar archive.

tar: log-type3/log-type3-2011-11-15.log: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.

I solved this issue by using the wonders bash expansion, sub-shell, and std error redirection
bash$ tar cfz yesterday-logs.tgz $( ls log-type{1,2,3,4}/*-2011-11-15.log} 2>/dev/null )
Fun. I suppose there is most likely a flag to tar that I couldn't find in my scan of the manual entry that completely obviate this complexity, but in the meantime...

Sunday, October 17, 2010

Update Ruby Gems - CentOS

Circular dependencies suck. It's a really great thing that the entire package management system for Ruby is written with Ruby, until you need to update Ruby using said system when said system itself needs to be updated.

I'm currently trying to update Ruby and RubyGems on CentOS via yum and life has not been fun. Step 1, rubygems wasn't present, so I installed with yum. That went fine, except it installed version 0.9.2 of gems. Hmm, Ok, no big deal, I can run gem update --system. Nope, fail:

me@host$ sudo gem update ruby
Updating installed gems...
ERROR:  While executing gem ... (Gem::RemoteSourceException)
    HTTP Response 302

Great, rubygems changed their URL since 0.9.2 and apparently whatever http tool gems uses can't follow redirects. Well, after some debugging and total failure to find any help online, I discovered that gems maintains a sources file /usr/lib/ruby/gems/1.8/gems/sources-0.0.1/lib/sources.rb that lists the domain(s) of its repositories. Well, let me just change that to the new URL

module Gem
  @sources = [""]
  def self.sources

I ran the command again. STILL getting the 302. What the heck??? Alright, after some more debugging, I discover it's trying to download a yaml description file located at This loads fine in my browser, but a little telnet shows that I'm being redirected (obvious hint from the 302 earlier).

[me@host]$ telnet 80
Connected to (
Escape character is '^]'.
GET /yaml HTTP/1.1

HTTP/1.1 302 Found
Date: Mon, 18 Oct 2010 05:46:40 GMT
Server: Apache/2.2.3 (Red Hat) mod_ssl/2.2.3 OpenSSL/0.9.8e-fips-rhel5 Phusion_Passenger/2.2.15
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 2.2.15
X-UA-Compatible: IE=Edge,chrome=1
X-Runtime: 0.000895
Content-Length: 0
Status: 302
Vary: Accept-Encoding
Content-Type: text/html

Hey, look! There's the redirect URL! A little update and I'm good to go.

module Gem
  @sources = [""]
  def self.sources

Then run,
[me@host]$sudo gem update --system
Updating RubyGems...
Attempting remote update of rubygems-update
Successfully installed rubygems-update-1.3.6
Installing ri documentation for rubygems-update-1.3.6...
Installing RDoc documentation for rubygems-update-1.3.6...
Could not find main page README
Could not find main page README
Could not find main page README
Could not find main page README
Updating version of RubyGems to 1.3.6
Installing RubyGems 1.3.6
ERROR:  Expected Ruby version >= 1.8.6, is 1.8.5
RubyGems system software updated

### Damn
### Maybe update ruby with gem? 

[me@host]$ sudo gem update ruby
Updating installed gems...
Attempting remote update of ruby
ERROR:  While executing gem ... (Gem::GemNotFoundException)
    Could not find ruby (> 0) in any repository

Nope, no go. So it looks like I am manually installing Ruby from the source since the yum repository only has Ruby as modern as version 1.8.5 dated 2006-08-25!! Isn't Ruby and Rails supposed to make life easier??

Friday, September 24, 2010

HTML5 Drag and Drop Upload Testing

We're building out a testing automation framework at work and one of the most important parts of it involves tests around file uploads. Historically this has been impossible to automate due to security settings in the browser that disallow setting the file path in a file type input element(e.g. to prevent secretly uploading confidential data from client machines). However, some browsers now support configurations to allow disabling this restriction for certain websites. Test automation driver, Selenium, has taken advantage of this and allows a user to set the value of a file type input element. This works great for the simple case of an explicit form and file type input element.

Enter HTML5. Browsers can now respond to the "drop" event and furthermore the XmlHttpRequest specification ( has been enhanced to make it possible to programmatically upload a file with javascript (i.e. xhr.send(fileBlob)). Put those two together and an ambitious web developer can now add automatic drag and drop file uploading to their website.

This is great for the look and feel of a website, but presents a challenge for the tester: How do I simulate a file drag and drop event? We were able to solve this issue by skipping the "drag" part and jumping straight to "drop." What does this mean? It means that our test will automatically call the "drop" event handler and provide a mock javascript event to the handler. The handler takes care of creating the xhr and sending it.

There is still one hurdle to overcome. The xhr requires an actual File object ( Since it's not possible to create one via standalone javascript, we got around this by creating a new file type input on the page and using Selenium to set its value (a path to a valid file on disk) and then referencing that.

The following Scala snippet uses jQuery for the dirty work:

val s = selenium.getEval("fileUpload1 = window.$j('<input/>').attr({type:'file'}).appendTo('body');")


// Trigger the upload event
val mockEventJs = "{originalEvent : {dataTransfer : { files : fileUpload1.get(0).files } } }"

val ss = selenium.getEval("window.uploader.handle_drop(" + mockEventJs + ")")

Saturday, September 18, 2010

Arel and Rails 3 - The Rage and Grace

I've been spending my time recently rebuilding the front end of a logging system we have to use Rails 3. This has been a lot of fun and I've come up with some fun code (pagination and naturaldate). The new ActiveRecord::Relation class has really made my data interactions streamlined and readable -- including (and especially) performant.

Unfortunately, I've found that Arel and Rails 3's journey into SQL's GROUP BY clause was a little brief. While I appreciate ActiveSupport::OrderedHash as a combination of array and hash to effectively express the results of a group by -- and effectively any other such ordered hash data set -- I was frustrated and disappointed by the way it handles multiple properties in the GROUP BY or SELECT clause. Additionally, I was thrown off for some time before I realized the fact that :count() returns an OrderedHash and not a number when dealing with a :group type active relation. Perhaps it's my own inability to figure out how to use Rails' or Arel's (poorly (if at all) documented) methods, but I've found that if I want any information outside of a basic

SELECT COUNT(*), property_name FROM tbl GROUP BY property_name

that I am forced to do more than one query. Hmm, I thought ORM was supposed to make my life easier or at least not make my website slower?

In my specific case, I need to group by several properties and paginate over the results. On top of that, I also needed an aggregate result in my select list. Not to mention the pagination parts.

SELECT prop1, prop2, COUNT(prop2) FROM tbl GROUP BY prop1, prop2

This is how I solved it:

counts = Model.
select("prop1, prop2, COUNT(prop2) AS 'prop3').
group("prop1, prop2")

num_pages = counts.all.size

page = counts.
offset(page_num * page_size)

There are two important things to notice above as hacks/sadness
  1. counts.all.size -- This will query the full set from the database as opposed to COUNT(*)
  2. COUNT(prop2) AS 'prop3' -- Hmm, what? Since rails will ignore anything in the select list that isn't a property on the referenced model's table, I have to trick into loading my aggregation result into prop3. Ugly.
Part 2:

Now that all of that is behind me, I'm faced with a new problem! Rails 3 doesn't support :include with :group! Oh pain, why can't life be simple? How did I solve this? Well, more hackery.

include() works by creating an instance of the included class and then uses its :find method to load all the referenced records after the initial model's data has been loaded in what might be called lazy loading, but what seems more like 1 + 1 versus N + 1 loads. Rails is able to programmatically do this in an elegant fashion by loading a reflections property on every ActiveRecord that defines a set of all foreign key relations to the singleton of that foreign key's ActiveRecord (Model) class.

Using this knowledge, I can imitate that same behavior by collecting all the referenced foreign keys in my select class (into :ids) and do the following,

fk1_records =[fk1_class_name.to_sym].klass.find(ids)

I can then iterate over the result set and define the relationships accordingly.

Tuesday, July 27, 2010

mapper.rb:143:in `default_controller_and_action': missing :action (ArgumentError)

I'm trying out Rails 3 for a small web project we need done at work. After spending more time than I wanted yesterday getting everything configured correctly on my Mac (namely mysql 64 bit), I was finally ready to start generating some MVC action. I followed the steps on the Rails 3 intro page to generate a controller for my home index. This added a route to my routes.rb,

get "home/index"

I thought I'd get fancy also add a root controller and view for my website, so I coped that and slightly modified it to

root :to => "home/index"

Cool. Next, I went ahead and tried to generate a new scaffold for my first model, but uh oh, there was a problem!

/Library/Ruby/Gems/1.8/gems/actionpack-3.0.0.beta4/lib/action_dispatch/routing/mapper.rb:143:in `default_controller_and_action': missing :action (ArgumentError)
from /Users/.../Code/.../config/routes.rb:2
from /Library/Ruby/Gems/1.8/gems/railties-3.0.0.beta4/lib/rails/commands.rb:16
from script/rails:6:in `require'
from script/rails:6

My first reaction was, "!@# yet another problem with Rails 3 and my system," but after some searching for the error turned up nothing, I decided that I had probably screwed something up. Looking a little more closely at the stack trace of the error, I identified that the scaffold generation was in part based on code from my own app and not just the rails gems. Hmm, probabilities rising that I'm the culprit.

The error says that the default controller is missing an action. I checked out the gem, mapper.rb, and saw that it's parsing out some variable named "to" for a controller and action. I then opened up my routes.rb and saw that line of copied code, root :to => "home/index". I then thought back to the new rails syntax that I'd learned in the video tutorials and immediately realized my folly. One quick change,

root :to => "home#index"

and the generator is back to working. Although it can definitely be a pain dealing with open source software, it certainly is nice to be able to see the source of your problems.

Wednesday, July 7, 2010

Migrating from svn to git

Several tutorials have already documented the necessary steps for the conversion from svn to git (, so I won't reiterate any of that. What made our migration tricky were two things:
  1. SVN commits without authors
  2. No tags, branches, or trunk paths
The second was easy to fix actually. I didn't specify any flags for standard layout or tags/branches in the git svn clone command. The second took a little bit of guess work. Before I begin, I'll point out that I used the script (after cleaning up the leading and trailing white space)

svn -q log | grep ^r | cut -d '|' -f 2 | sort | uniq > authors.txt

as conveniently provided in the comments to one of the tutorials. Just to make life easy, I used the following VI macro to fill out the definitions in the file,

yeA= ^[pa <^[>^[j0

I noticed a blank line in the file, which was generated because of commits to the repo with no user name -- literally just the empty string. Big deal, I just removed it. When I tried to run the "git svn clone" though, it puked when it got to those revisions:

$>git svn clone svn://svn.domain/project -A authors.txt
Author: (no author) not defined in authors.txt file

I googled and found a support article,, which didn't actually solve the problem, just avoided it. It turned out the fix was actually easier than I thought. I added the following line to the authors file:

(no author) = no_author

Tuesday, March 16, 2010

IdeaVIM in IntelliJ Community Edition

So, I started using IntelliJ last month and I've really been enjoying it. As a previous heavy user of ReSharper with Visual Studio, I've grown accustomed to the convenience and efficiency of an IDE with refactoring and code inspection built in. However, I immediately began to suffer from the lack of VI keybindings.

One of the great things about IntelliJ is the huge repository of plugins. To top that off, the plugins are integrated right into the preferences manager so you can search, download, and enable them all in one easy place. My search in the plugins manager came up empty, so I continued the search online where I did find a plugin listed right on the IntelliJ website! IdeaVIM:

Why wasn't it showing up in my plugins manager? Well, it turns out that the IdeaVIM plugin is only offered to users of the IntelliJ Ultimate edition. Great... I'm not about to pay $250 to get key bindings for one of the most prevalent text editors in the world in my IntelliJ, on top of the fact that the money doesn't go directly to the plug-in developer (Note, I was happy to pay $60 for my copy of ViEmu).

So two things happened next that got me what I needed,
  1. I saw a "download" link on the plug-ins page
  2. I learned that my coworker had successfully installed the plugin on his computer before they restricted it to the Ultimate edition
With that knowledge in hand, I was ready to try a new tactic:
  1. I downloaded and unzipped the plugin and placed it in my ~/Library/Application Support/IntelliJIdea90CE/ directory.
  2. Next, I copy-and-pasted the availables.xml configuration section for the IdeaVIM plugin off my coworker's computer into my own.
  3. Then I restarted IntelliJ and ta-da, I now have VIM!

Wednesday, January 20, 2010

Where Does Outlook Keep My Email?

My web searches on this topic were helpful, but incomplete. Our company uses an exchange server to house all of our corporate emails, so technically all of my email is on the exchange server. For performance, however, my email is also stored locally on my machine in my user's local settings: C:\Documents and Settings\%user%\Local Settings\Application Data\Microsoft\Outlook. (Don't forget the local settings part.) This much info I found online.

The complicated part is that my company just switched to exchange -- I'm not sure what we were using prior -- so I have more than one PST (personal storage) in there. On a hunch, I checked out my control panel's "Mail" interface.
  1. Open Control Panel -> Mail
  2. Click E-mail Accounts
  3. Switch to tab "Data Files"
  4. The path to the PST file for my current profile is right there!
That was easy.