1961: Git SSH repository: Cannot spawn rbssh

Anthony*******@gmai***** (Google Code) (Is this you? Claim this profile.)
chipx86
chipx86
Feb. 7, 2011
What version are you running?
1.5.2

What's the URL of the page containing the problem?
http://rbserver/admin/db/scmtools/repository/#/

What steps will reproduce the problem?
1. Choose custom hosting service
2. Repository type git
3. Define a path with ssh

What is the expected output? What do you see instead?
I expect that the repository is successfully loaded.  Instead, I see an error "A repository was not found at the specified path."  There is an error in the RB logs about not being able to spawn rbssh.  The log indicates that it was able to login with the provided credentials before it fails to spawn rbssh.

2011-01-26 16:08:52,545 - DEBUG - GitTool: Attempting ssh connection with host: xxx.xxx.com, username: joeuser
2011-01-26 16:08:52,559 - DEBUG - starting thread (client mode): 0x6610590L
2011-01-26 16:08:52,559 - INFO - Connected (version 2.0, client OpenSSH_5.2)
2011-01-26 16:08:52,591 - DEBUG - kex algos:['diffie-hellman-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa', 'ssh-dss'] client encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijndael-cbc@lysator.liu.se'] server encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijndael-cbc@lysator.liu.se'] client mac:['hmac-md5', 'hmac-sha1', 'umac-64@openssh.com', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-sha1-96', 'hmac-md5-96'] server mac:['hmac-md5', 'hmac-sha1', 'umac-64@openssh.com', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['none', 'zlib@openssh.com'] server compress:['none', 'zlib@openssh.com'] client lang:[''] server lang:[''] kex follows?False
2011-01-26 16:08:52,591 - DEBUG - Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
2011-01-26 16:08:52,591 - DEBUG - using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none
2011-01-26 16:08:52,763 - DEBUG - Switch to new keys ...
2011-01-26 16:08:52,779 - DEBUG - Trying discovered key xxxxxxxxxxxxxxxxxxxxxxxxxx in C:\Documents and Settings\joeuser2/.ssh/id_rsa
2011-01-26 16:08:52,809 - DEBUG - userauth is OK
2011-01-26 16:08:53,309 - INFO - Authentication (publickey) failed.
2011-01-26 16:08:53,325 - DEBUG - userauth is OK
2011-01-26 16:08:53,357 - DEBUG - Authentication type (password) not permitted.
2011-01-26 16:08:53,357 - DEBUG - Allowed methods: ['publickey', 'gssapi-keyex', 'gssapi-with-mic', 'keyboard-interactive']
2011-01-26 16:08:53,372 - DEBUG - userauth is OK
2011-01-26 16:08:54,342 - INFO - Authentication (keyboard-interactive) successful!
2011-01-26 16:08:54,434 - DEBUG - EOF in transport thread
2011-01-26 16:08:54,467 - ERROR - Git: Failed to find valid repository ssh://joeuser@xxx.xxx.com/somerepo.git: error: cannot spawn rbssh: No such file or directory
fatal: unable to fork

What operating system are you using? What browser?
2K8 Server R2 Standard, Firefox

Please provide any additional information below.
I chose to have RB generate its own key, but I notice that it did not use that generated key and instead tried to use a default user key, which wasn't even the user's that was logged in.  I find it strange that it was able to authenticate in interactive mode, but then claims to fail to find a valid repository complaining about running rbssh ?
chipx86
#1 chipx86
This looks to me like there are a couple problems.

First, your Apache configuration likely was never updated to set the HOME variable. It should be set to {yoursitedir}\data (make sure this directory exists and is writeable by the web server). The generated key should go there.

Second, there may be some issue on Windows launching rbssh. Or it may be local to your system. I assume you can run rb-site from your user for the upgrade. Can you manually run rbssh?

If you can, it's possible that your system-wide path isn't taking into account the directory where rbssh lives. You'll need to update that and reboot.
  • +NeedInfo
#2 Anthony*******@gmai***** (Google Code) (Is this you? Claim this profile.)
Thanks for writing back quickly!
I see now that another user had overridden the system path to point at a different python scripts directory, doh.
Now rbssh can load, but now rbssh-script.py fails with an import error on the module pwd.

ImportError: No module named pwd

Where can I get this module?  Perhaps it should be added in the upgrade process
#3 Anthony*******@gmai***** (Google Code) (Is this you? Claim this profile.)
2011-01-26 17:28:23,963 - ERROR - Git: Failed to find valid repository ssh://xxx.xxx.com/somerepo.git: D:\Python25\lib\site-packages\Crypto\Util\randpool.py:40: RandomPool_DeprecationWarning: This application uses RandomPool, which is BROKEN in older releases.  See http://www.pycrypto.org/randpool-broken

  RandomPool_DeprecationWarning)

Traceback (most recent call last):

  File "D:\Python25\Scripts\rbssh-script.py", line 8, in <module>

    load_entry_point('ReviewBoard==1.5.2', 'console_scripts', 'rbssh')()

  File "d:\python25\lib\site-packages\ReviewBoard-1.5.2-py2.5.egg\reviewboard\cmdline\rbssh.py", line 205, in main

    client.connect(hostname, username=username)

  File "build\bdist.win32\egg\paramiko\client.py", line 319, in connect

  File "D:\Python25\lib\getpass.py", line 106, in getuser

    import pwd

ImportError: No module named pwd
#4 Anthony*******@gmai***** (Google Code) (Is this you? Claim this profile.)
P.S. I have pycrypto 2.1 installed for python 2.5
chipx86
#5 chipx86
pwd is a Linux-only Python module, as far as I can tell. It looks like Python assumes it can access it, even though it can't. This is outside our control.

Something you can try for getting around this is to make the username part of the SSH URL. It will then parse it out and include it, bypassing the call to getuser.
#6 Anthony*******@gmai***** (Google Code) (Is this you? Claim this profile.)
Ah, good idea.  I made my username part of the SSH URL now and it is able to skip over the pwd call.
It now complains that the authentication failed.  It appears that it is trying to connect with just a hostname and username, no password, even though a password was specified in the repository configuration form.  It was able to log in successfully in keyboard-interactive mode with the given credentials. 

Does this mean that there's a conflict of authentication method since I tried to generate a SSH key AND I provided login credentials?  If so, is it possible for me to force credentials or undo the SSH key?
chipx86
#7 chipx86
It *should* try both password and public key.

Did you try re-generating/uploading a key since making the %HOME% change? If so, it no longer knows about that key.
#8 Anthony*******@gmai***** (Google Code) (Is this you? Claim this profile.)
I tried uploading a key but RB is trying to load the key from the home directory still.  
#9 Anthony*******@gmai***** (Google Code) (Is this you? Claim this profile.)
Ahh.. generate_user_key in sshutils.py just returns the found key in the home directory even if a key is uploaded.
#10 Anthony*******@gmai***** (Google Code) (Is this you? Claim this profile.)
OK so now I have public key authentication working OK.  In the reviewboard logs it discovers the correct key and indicates that authentication with the public key is successful.
Immediately after it successfully logs in with the pubkey, it raises a debug message EOF in transport thread and the reviewboard website fails to respond to the save repository call.

So far I think that it'd be useful to indicate in the SSH settings that if you try to generate a key that it won't override the existing key.  I'm not sure how I ended up at the choice to generate a key because I believe that it's supposed to show you the current key instead and not give you a choice to generate/upload a key.

I'm trying to debug the timeout problem now
#11 Anthony*******@gmai***** (Google Code) (Is this you? Claim this profile.)
After I hit save to save the ssh git repository, the SSH session just seems to die for some reason and then RB times out.  Here's the RB log.

2011-01-31 15:35:03,733 - DEBUG - GitTool: Attempting ssh connection with host: xxx.xxxx.com, username: joeuser
2011-01-31 15:35:03,733 - DEBUG - starting thread (client mode): 0x5caf250L
2011-01-31 15:35:03,795 - INFO - Connected (version 2.0, client OpenSSH_5.2)
2011-01-31 15:35:03,826 - DEBUG - kex algos:['diffie-hellman-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa', 'ssh-dss'] client encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijndael-cbc@lysator.liu.se'] server encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijndael-cbc@lysator.liu.se'] client mac:['hmac-md5', 'hmac-sha1', 'umac-64@openssh.com', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-sha1-96', 'hmac-md5-96'] server mac:['hmac-md5', 'hmac-sha1', 'umac-64@openssh.com', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['none', 'zlib@openssh.com'] server compress:['none', 'zlib@openssh.com'] client lang:[''] server lang:[''] kex follows?False
2011-01-31 15:35:03,826 - DEBUG - Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
2011-01-31 15:35:03,826 - DEBUG - using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none
2011-01-31 15:35:03,999 - DEBUG - Switch to new keys ...
2011-01-31 15:35:03,999 - DEBUG - Trying discovered key 2f58fb0653805cffea0cdec82455c096 in D:\ReviewBoard\data/.ssh/id_rsa
2011-01-31 15:35:04,029 - DEBUG - userauth is OK
2011-01-31 15:35:04,686 - INFO - Authentication (publickey) successful!
2011-01-31 15:35:04,795 - DEBUG - EOF in transport thread

chipx86
#12 chipx86
For the key, it specifically looks for id_dsa or id_rsa. It sounds like maybe it was still either looking in the wrong location, or you manually copied a file there?

Anyway, the SSH session isn't supposed to be up for too long, as it just runs a small command to get some info, but it seems this is failing. It's probably something specific to the Windows implementation of rbssh, as getting this right on Windows is far, far more tricky than on Linux/Mac, unfortunately.

I have no immediate answer. Until I get a chance to debug this on a Windows system and see if I can reproduce it, the only thing I can suggest is to modify rbssh to do some debugging (you can set DEBUG to True in there and add additional logging with debug("sometihng")), or, ideally, move your server to Linux. Review Board works far better in many ways on Linux than it does on Windows, due to the many dependencies we use being primarily tested on Linux.
#13 Anthony*******@gmai***** (Google Code) (Is this you? Claim this profile.)
I couldn't get too much info the pipe just terminates prematurely without a good reason.

I set up a new reviewboard instance on an ubuntu vm and everything works swimmingly.  Thanks for the info though hopefully other people don't run into this too and I'll remember to start with a linux distro for future use :)
chipx86
#14 chipx86
Glad it works now.

I've spent the last few days working on ironing out the remaining kinks in rbssh and putting together a whole testing infrastructure to make sure this doesn't break again. I'm hoping to have a release out in a few days.
  • -NeedInfo
    +Confirmed
  • -Priority-Medium
    +Priority-Critical
    +Component-SSH
    +Milestone-Release1.5.x
  • +chipx86
chipx86
#15 chipx86
Fixed on release-1.5.x and master. This will be in for a 1.5.3 release shortly.
  • -Confirmed
    +Fixed