Wednesday, October 26, 2011

Gerrit Code Review - Unpack error Missing unknown

We use the Gerrit code review tool at my company; currently version 2.1.8. Gerrit gives us a lot of tools and is an extremely useful tool for code quality and knowledge sharing (i.e. code reviewing). However, Gerrit can often require some TLC. Today was one of those days.

While I was doing some maintenance on our git host today, I noticed that git suggested I prune one of our repos. So, I went ahead and pruned. About half an hour later, one of our developers reported that he couldn't create a new patch set in Gerrit -- neither update or create were working. Pretty soon, I was getting several reports of this from the team. They were all getting the same error:


error: unpack failed: error Missing unknown 3061766be9c324fa47fb4832399b34db5a186276
fatal: Unpack error, check server log
To ssh://git.dev:29418/webapp
 ! [remote rejected] HEAD -> refs/for/master/master (n/a (unpacker error))

The key part of this message is the git object that's missing, 3061766. The git prune must have identified that the object was dangling, no longer necessary, and removed it. However, for some reason, JGit inside of Gerrit still felt the object served some purpose important enough to throw an exception. I tried to find the object in the remote repo and locally, but to no avail, it was history.

After trying pretty much everything I could find on the internet (http://code.google.com/p/gerrit/issues/detail?id=585http://groups.google.com/group/repo-discuss/browse_thread/thread/cf7095d3dc364c7e/f2c11756a5a0396f?) and replacing the jgit jar in my tmp .gerritcodereview directory, I finally found a solution.

The solution was to restore a backup of our git repository to a temp directory and use git cat-file to first verify the object 3061766 was valid. Then, I used git verify-pack to find the pack file that contained the commit. Then I copied the pack and its index file (http://progit.org/book/ch9-4.html) into our real repo. Shazzam! Gerrit started accepting change sets again!

In more detail:

  1. tar -xf backup.tar ./path-to-git-dir
  2. cd path-to-git-dir
  3. git cat-file -t 3061766be9c324fa47fb4832399b34db5a186276
  4. cd objects/pack
  5. ls | xargs git verify-pack -v | grep 3061766be9c324fa47fb4832399b34db5a186276
  6. # back track until you find the pack that contains the commit
  7. cd /git/path-to-git-dir/objects/pack
  8. cp /untarred-repo/objects/pack/...{idx,pack} .



4 comments:

  1. Was there an overlap in the packs? i.e. did you find the object in more than one pack? And/or did your original repo already have a pack by that hash? And maybe some of the objects inside the new pack were duplicates? Did that cause any problems?

    ReplyDelete
  2. Yeah, I found commit object in two of the packs. I copied the older pack back into our repo since I decided its probability of having more accurate information was higher. My understanding is also that pack names are unique by the same guarantee of all tree-ish references.

    ReplyDelete
  3. When we found a similar issue, I went into the Gerrit database and removed references to the missing commit.

    ReplyDelete
  4. Thanks for the tip.

    find . -name "*.pack" -exec sh -c "git verify-pack -v {} | grep c30dc6604e68e0179fc8c2358d023572307b6308 && echo {}" \;

    Will find the actual pack file.

    ReplyDelete