Sunday, May 4, 2014

Debugging Gitosis "Read Access Denied"

I recently set up gitosis to serve some side projects that I'd like to share with a few friends. I've used it in the past professionally and really enjoy the sanity it brings to managing users and permissions.

Things started off pretty well and I was committing and pushing changes in no time. A week or so passed and I wanted to add a new user to a project. I made the necessary changes to my clone of the gitosis-admin project, but when I tried to push my changes upstream, I suddenly I was unable to push! This was a major issue since the admin project is the heart of the configuration.

I put on my spelunking hat and ssh-ed to the box and switched to the git user to start debugging. The first thing I did was revert the gitosis.conf file back to it's original state. You can find this file in ~/git/repositories/gitosis-admin.git/gitosis.conf. Changing it had no effect.

I took a closer look at the error message from my failed push command, and noticed that it was complaining "Read Access Denied," but for a different user name (I could see this because I had loglevel = DEBUG). There are a total of three users involved in the projects, one of which I'd just added and only locally. So, on the server, there were only two users at play. OK, so maybe that user is causing issues. I next removed his key file from the server. This file was at ~git/repositories/gitosis-admin.git/gitosis-export/keydir/.

I tried to push again. No luck.

Hmm, next I looked into ~/git/.ssh/authorized_keys. I found there was a still a reference to the user there, so I deleted that line.

I tried to push again. It worked!

Ok, so are things working now? I tried to fetch. No dice.

So, when I pushed, gitosis re-applied the configuration and undid all of my debugging steps. Essentially reverting the system back to the previous state, including new additions for the new user.

At this point, it dawned on me to check my ssh agent identities. Lo and behold, I had two identities and one of them was for the other user! Oops! This was completely my mistake. I had generated his keys a few weeks ago and tested them to ensure they worked. Apparently I had not been so thoughtful as to delete the identity when done.

After running ssh-add -D, things started working again.