Gerrit Code Review - Unpack error Missing unknown
We use the Gerrit code review tool at my company; currently version 2.1.8. Gerrit gives us a lot of tools and is an extremely useful tool for code quality and knowledge sharing (i.e. code reviewing). However, Gerrit can often require some TLC. Today was one of those days.
While I was doing some maintenance on our git host today, I noticed that git suggested I prune one of our repos. So, I went ahead and pruned. About half an hour later, one of our developers reported that he couldn't create a new patch set in Gerrit -- neither update or create were working. Pretty soon, I was getting several reports of this from the team. They were all getting the same error:
error: unpack failed: error Missing unknown 3061766be9c324fa47fb4832399b34db5a186276
fatal: Unpack error, check server log
To ssh://git.dev:29418/webapp
! [remote rejected] HEAD -> refs/for/master/master (n/a (unpacker error))
The key part of this message is the git object that's missing, 3061766. The git prune must have identified that the object was dangling, no longer necessary, and removed it. However, for some reason, JGit inside of Gerrit still felt the object served some purpose important enough to throw an exception. I tried to find the object in the remote repo and locally, but to no avail, it was history.
After trying pretty much everything I could find on the internet (http://code.google.com/p/gerrit/issues/detail?id=585, http://groups.google.com/group/repo-discuss/browse_thread/thread/cf7095d3dc364c7e/f2c11756a5a0396f?) and replacing the jgit jar in my tmp .gerritcodereview directory, I finally found a solution.
The solution was to restore a backup of our git repository to a temp directory and use git cat-file to first verify the object 3061766 was valid. Then, I used git verify-pack to find the pack file that contained the commit. Then I copied the pack and its index file (http://progit.org/book/ch9-4.html) into our real repo. Shazzam! Gerrit started accepting change sets again!
In more detail:
While I was doing some maintenance on our git host today, I noticed that git suggested I prune one of our repos. So, I went ahead and pruned. About half an hour later, one of our developers reported that he couldn't create a new patch set in Gerrit -- neither update or create were working. Pretty soon, I was getting several reports of this from the team. They were all getting the same error:
error: unpack failed: error Missing unknown 3061766be9c324fa47fb4832399b34db5a186276
fatal: Unpack error, check server log
To ssh://git.dev:29418/webapp
! [remote rejected] HEAD -> refs/for/master/master (n/a (unpacker error))
The key part of this message is the git object that's missing, 3061766. The git prune must have identified that the object was dangling, no longer necessary, and removed it. However, for some reason, JGit inside of Gerrit still felt the object served some purpose important enough to throw an exception. I tried to find the object in the remote repo and locally, but to no avail, it was history.
After trying pretty much everything I could find on the internet (http://code.google.com/p/gerrit/issues/detail?id=585, http://groups.google.com/group/repo-discuss/browse_thread/thread/cf7095d3dc364c7e/f2c11756a5a0396f?) and replacing the jgit jar in my tmp .gerritcodereview directory, I finally found a solution.
The solution was to restore a backup of our git repository to a temp directory and use git cat-file to first verify the object 3061766 was valid. Then, I used git verify-pack to find the pack file that contained the commit. Then I copied the pack and its index file (http://progit.org/book/ch9-4.html) into our real repo. Shazzam! Gerrit started accepting change sets again!
In more detail:
- tar -xf backup.tar ./path-to-git-dir
- cd path-to-git-dir
- git cat-file -t 3061766be9c324fa47fb4832399b34db5a186276
- cd objects/pack
- ls | xargs git verify-pack -v | grep 3061766be9c324fa47fb4832399b34db5a186276
- # back track until you find the pack that contains the commit
- cd /git/path-to-git-dir/objects/pack
- cp /untarred-repo/objects/pack/...{idx,pack} .
Comments
find . -name "*.pack" -exec sh -c "git verify-pack -v {} | grep c30dc6604e68e0179fc8c2358d023572307b6308 && echo {}" \;
Will find the actual pack file.