Bug 39373 - scp task does not work for some sshd, which does not scp1 protocol.
scp task does not work for some sshd, which does not scp1 protocol.
Status: RESOLVED FIXED
Product: Ant
Classification: Unclassified
Component: Optional Tasks
1.6.5
All other
: P2 normal with 1 vote (vote)
: 1.7.0
Assigned To: Ant Notifications List
:
: 39532 (view as bug list)
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2006-04-21 01:27 UTC by Atsuhiko Yamanaka
Modified: 2008-07-14 06:44 UTC (History)
3 users (show)



Attachments
a patch to use sftp protocol in scp task. (20.11 KB, patch)
2006-04-21 01:37 UTC, Atsuhiko Yamanaka
Details | Diff
an implementation of sftp task (119.49 KB, patch)
2006-04-26 15:16 UTC, Atsuhiko Yamanaka
Details | Diff
Patch to scp.html manual page (707 bytes, patch)
2006-05-09 17:12 UTC, Robert Anderson
Details | Diff
bugfix to download files/directory, which are pointed by symbolic link. (1.87 KB, patch)
2006-09-27 09:35 UTC, Atsuhiko Yamanaka
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Atsuhiko Yamanaka 2006-04-21 01:27:56 UTC
In the current implementaion, scp task included in Apache Ant has been using scp1
protocol to transfer files, but some proprietary sshd[1] has not supported it anymore.

To resolve this problem, scp task should use sftp protocol[2], which must be supported by
almost of all ssh2 sshd.

[1] http://www.snailbook.com/faq/scp-ossh-to-ssh2.auto.html
[2] http://www.unix.org.ua/orelly/networking_2ndEd/ssh/ch03_08.htm
Comment 1 Atsuhiko Yamanaka 2006-04-21 01:37:34 UTC
Created attachment 18147 [details]
a patch to use sftp protocol in scp task.

Attached patch will allow scp task to use sftp protocol instead of scp1
protocol.

If you add the attribute 'sftp="true"', sftp protocol will be used and 
if 'sftp="false"', scp1 protocol will be used.	
By the default, the attribute 'sftp' is "false" for backword compatibity.

To try this patch, jsch-0.1.27[1] is required.

[1] http://sourceforge.net/project/showfiles.php?group_id=64920
Comment 2 Benjamin Burgess 2006-04-21 18:17:01 UTC
Wouldn't it be better to define a new task called "sftp" rather than changing
the "scp" task to no longer use scp?
Comment 3 Jeffrey E. Care 2006-04-21 21:05:08 UTC
+1 on Ben's comment
Comment 4 Robert Anderson 2006-04-21 21:28:29 UTC
-1 on Ben's comment.

Please read the links provided by Atsuhiko, particularly
[2] http://www.unix.org.ua/orelly/networking_2ndEd/ssh/ch03_08.htm. 

SFTP is the protocol used by the program scp2. It is entirely appropriate to 
add this feature to the scp task.
Comment 5 Robert Anderson 2006-04-22 00:35:40 UTC
After applying the patch and running a basic test against an F-Secure sshd I 
get the following error:

BUILD FAILED
H:\build.xml:6: com.jcraft.jsch.JSchException: invalid server's version string

I am using the jsch 0.1.27
Comment 6 Atsuhiko Yamanaka 2006-04-24 03:29:52 UTC
(In reply to comment #5)
> After applying the patch and running a basic test against an F-Secure sshd I 
> get the following error:
> BUILD FAILED
> H:\build.xml:6: com.jcraft.jsch.JSchException: invalid server's version 
string
> I am using the jsch 0.1.27

Does that sshd support ssh2 protocol?
Comment 7 Robert Anderson 2006-04-24 05:23:23 UTC
(In reply to comment #6)
> (In reply to comment #5)
> > After applying the patch and running a basic test against an F-Secure sshd 
I 
> > get the following error:
> > BUILD FAILED
> > H:\build.xml:6: com.jcraft.jsch.JSchException: invalid server's version 
> string
> > I am using the jsch 0.1.27
> Does that sshd support ssh2 protocol?

The version of F-Secure sshd we are using supports only ssh2 protocol.
Comment 8 Atsuhiko Yamanaka 2006-04-24 05:43:55 UTC
(In reply to comment #7)
> The version of F-Secure sshd we are using supports only ssh2 protocol.

It is interesting.
May I ask you to show me the fist line of outputs from

  $ telnet address_of_sshd 22
Comment 9 Robert Anderson 2006-04-24 16:45:34 UTC
(In reply to comment #8)
> (In reply to comment #7)
> > The version of F-Secure sshd we are using supports only ssh2 protocol.
> It is interesting.
> May I ask you to show me the fist line of outputs from
>   $ telnet address_of_sshd 22

SSH-2.0-3.2.9 F-SECURE SSH 3.2.3
Comment 10 Robert Anderson 2006-04-24 22:55:27 UTC
(In reply to comment #9)
> (In reply to comment #8)
> > (In reply to comment #7)
> > > The version of F-Secure sshd we are using supports only ssh2 protocol.
> > It is interesting.
> > May I ask you to show me the fist line of outputs from
> >   $ telnet address_of_sshd 22
> SSH-2.0-3.2.9 F-SECURE SSH 3.2.3

It appears that the ascii codes for the server version string returned by F-
Secure is lower case. The following patch to Session.java will address the 
issue.

--- Session.java.orig   2006-04-18 22:37:02.000000000 -0700
+++ Session.java        2006-04-24 15:42:58.594132100 -0700
@@ -224,11 +224,13 @@
        }
       }
 
-      if(i==buf.buffer.length ||
+      if((i==buf.buffer.length ||
          (i>4 && (buf.buffer[0]!='S'||buf.buffer[1]!='S'||
                   buf.buffer[2]!='H'||buf.buffer[3]!='-'))||
         i<7 ||                                      // SSH-1.99 or SSH-2.0
-        (buf.buffer[4]=='1' && buf.buffer[6]!='9')  // SSH-1.5
+        (buf.buffer[4]=='1' && buf.buffer[6]!='9')) ^ // SSH-1.5
+        (buf.buffer[0]!='s'||buf.buffer[1]!='s'||
+             buf.buffer[2]!='h'||buf.buffer[3]!='-') // "ssh-" F-Secure returns
 lowercase
         ){
        throw new JSchException("invalid server's version string");
       }

The task is still not working however. Now I get the following error...

com.jcraft.jsch.JSchException: Session.connect: java.io.IOException: invalid 
data

This exception is thrown during the Session.read(buf) method.
Comment 11 Atsuhiko Yamanaka 2006-04-25 00:09:08 UTC
(In reply to comment #10)
> (In reply to comment #9)
> > SSH-2.0-3.2.9 F-SECURE SSH 3.2.3
> 
> It appears that the ascii codes for the server version string returned by F-
> Secure is lower case. The following patch to Session.java will address the 
> issue.

It is curious.  You had written as in comment #9
SSH-2.0-3.2.9 F-SECURE SSH 3.2.3
, which starts with upper case string "SSH-" and the RFC4253[1] says at Section 4.2 as follows,

  4.2.  Protocol Version Exchange
  
     When the connection has been established, both sides MUST send an
     identification string.  This identification string MUST be

         SSH-protoversion-softwareversion SP comments CR LF

     Since the protocol being defined in this set of documents is version
     2.0, the 'protoversion' MUST be "2.0".  The 'comments' string is
     OPTIONAL.  If the 'comments' string is included, a 'space' character
     (denoted above as SP, ASCII 32) MUST separate the 'softwareversion'
     and 'comments' strings. 

I don't have the reason why it starts with 'ssh-'.

> The task is still not working however. Now I get the following error...
> com.jcraft.jsch.JSchException: Session.connect: java.io.IOException: invalid data
> This exception is thrown during the Session.read(buf) method.

If it really sents 'ssh-', it may not use formal ssh2 protocol, IMHO.
Anyway, that problem is not related to this topic.

[1] http://www.ietf.org/rfc/rfc4253.txt

Comment 12 Robert Anderson 2006-04-25 00:36:59 UTC
(In reply to comment #11)
> (In reply to comment #10)
> > (In reply to comment #9)
> > > SSH-2.0-3.2.9 F-SECURE SSH 3.2.3
> > 
> > It appears that the ascii codes for the server version string returned by F-
> > Secure is lower case. The following patch to Session.java will address the 
> > issue.
> It is curious.  You had written as in comment #9
> SSH-2.0-3.2.9 F-SECURE SSH 3.2.3
> , which starts with upper case string "SSH-" and the RFC4253[1] says at 
Section 4.2 as follows,
>   4.2.  Protocol Version Exchange
>   
>      When the connection has been established, both sides MUST send an
>      identification string.  This identification string MUST be
>          SSH-protoversion-softwareversion SP comments CR LF
>      Since the protocol being defined in this set of documents is version
>      2.0, the 'protoversion' MUST be "2.0".  The 'comments' string is
>      OPTIONAL.  If the 'comments' string is included, a 'space' character
>      (denoted above as SP, ASCII 32) MUST separate the 'softwareversion'
>      and 'comments' strings. 
> I don't have the reason why it starts with 'ssh-'.
> > The task is still not working however. Now I get the following error...
> > com.jcraft.jsch.JSchException: Session.connect: java.io.IOException: 
invalid data
> > This exception is thrown during the Session.read(buf) method.
> If it really sents 'ssh-', it may not use formal ssh2 protocol, IMHO.
> Anyway, that problem is not related to this topic.
> [1] http://www.ietf.org/rfc/rfc4253.txt

I agree that this is a seperate issue. I would like to work with you to resolve 
it, if possible. I am interested in the enhancements to the scp task and would 
like to see them committed to the Ant project. Should we take this discussion 
to the ant-dev list?
Comment 13 Atsuhiko Yamanaka 2006-04-26 15:16:05 UTC
Created attachment 18186 [details]
an implementation of sftp task

The attached patch is a quick hack to implement sftp task, which
is based on ftp task and using jsch for sftp functionality.
It accepts attributes for ftp and scp tasks except for 'binary',
'passive', 'separator' and 'umask'.  

It seems following tasks are working on my testing, however, frankly to say,
I'm not familiar with ftp task and have never used and run it, so I may
mis-understand its operational semantics.  Comments, suggestions and feedbacks
are welcome.

PS 1. that patch is for ant_20060426105634.tar.gz 
PS 2. jsch-0.1.27 is required.

  <sftp action="send"
       server="${server}"
       userid="${userid}"
       password="${password}"
       remotedir="incoming"
       depends="yes">
     <fileset dir="htdocs/manual"/>
  </sftp>

  <sftp action="get"
       server="${server}"
       userid="${userid}"
       password="${password}">
    <fileset dir="htdocs/manual" >
      <include name="**/*.html"/>
    </fileset>
  </sftp>

 <sftp action="del"
       server="${server}"
       userid="${userid}"
       password="${password}" >
    <fileset>
      <include name="**/*.tmp"/>
    </fileset>
  </sftp>

  <sftp action="list"
       server="${server}"
       userid="${userid}"
       password="${password}"
       listing="data/sftp.listing" >
    <fileset>
      <include name="**"/>
    </fileset>
  </sftp>

  <sftp action="mkdir"
       server="${server}"
       userid="${userid}"
       password="${password}"
       remotedir="test" />

  <sftp action="chmod"
       server="${server}"
       userid="${userid}"
       password="${password}"
       chmod="0700" >
    <fileset>
      <include name="*.log"/>
    </fileset>
  </sftp>
Comment 14 Robert Anderson 2006-04-26 16:29:00 UTC
Why the change in direction? I like the idea of adding the sftp attribute to 
the scp task, rather than creating a new sftp task. Perhaps both are a good 
idea.
Comment 15 Atsuhiko Yamanaka 2006-04-28 06:40:07 UTC
(In reply to comment #14)
> Why the change in direction? I like the idea of adding the sftp attribute to 
> the scp task, rather than creating a new sftp task. Perhaps both are a good 
> idea.

Of course, that patch includes both.
Comment 16 Robert Anderson 2006-05-02 23:13:58 UTC
After working out an issue with Jsch, I have successfully tested the 
enhancements to the scp task.

I think the sftp task enhancement should be in a seperate bug report, because I 
believe it is a different subject. Furthermore, I feel that this enhancement to 
the scp task is a higher priority than the need for a new sftp task.
Comment 17 atsuhiko.yamanaka 2006-05-03 00:21:35 UTC
(In reply to comment #5)
> After applying the patch and running a basic test against an F-Secure sshd I 
> get the following error:
> BUILD FAILED
> H:\build.xml:6: com.jcraft.jsch.JSchException: invalid server's version 
string
> I am using the jsch 0.1.27

Thanks to Robert, I found the problem in the jsch code and has fixed it, at 
last.
If somebody has simillar problem,  http://www.jcraft.com/jsch/jsch-0.1.28-
rc4.jar will resolve that problem.
Comment 18 Atsuhiko Yamanaka 2006-05-08 03:26:19 UTC
(In reply to comment #16)
> I think the sftp task enhancement should be in a seperate bug report, because I 
> believe it is a different subject. Furthermore, I feel that this enhancement to 
> the scp task is a higher priority than the need for a new sftp task.

I agree with him.  There should be a separate bug report for sftp task, but
that patch for sftp task is based on the source code for this modified scp task.

Dear ant developers,  are there not any possibilties that this modification for scp task will be adopted?
If so, I'll delete it from the source code and submit a new bug entry for sftp task
and post a patch, which does not include the modifications for scp.
I don't want to keep several source code trees.


Comment 19 Stefan Bodewig 2006-05-09 04:25:25 UTC
I've committed the part of the patch that adds sftp support to scp, the sftp
task will remain separate since it needs a bit more of review than I can do
right now.

You'll note that there are a few differences between your patch and what I
committed, most of them are whitespace only.

We also need an update of scp's manual page.  IIUC we should recommend to set
the sftp attribute to true if the ssh server supports ssh2, correct?
Comment 20 Robert Anderson 2006-05-09 17:12:22 UTC
Created attachment 18251 [details]
Patch to scp.html manual page
Comment 21 Atsuhiko Yamanaka 2006-05-10 03:48:48 UTC
(In reply to comment #19)
> We also need an update of scp's manual page.  IIUC we should recommend to set
> the sftp attribute to true if the ssh server supports ssh2, correct?

IMHO, if users have satisfied with original scp task, there are not any merit
to use sftp attribute, becase scp1 protocol is more efficent than sftp protocol.
So, I hesitate to recommend to use it explicitly.  That attribute is only for
some users, whose sshd does not support scp1 anymore.  Almost users will use
OpenSSH's sshd and it will continue to support scp1 for a while.

Comment 22 Stefan Bodewig 2006-05-11 04:02:00 UTC
(In reply to comment #20)

> Patch to scp.html manual page

I've applied a slightly modified version (recommending the option if the server
doesn't support SSH1 based on Atsuhiko Yamanaka's comment) but svn is down right
now so I can't commit it.
Comment 23 Atsuhiko Yamanaka 2006-05-11 06:56:48 UTC
(In reply to comment #22)
> I've applied a slightly modified version (recommending the option if the server
> doesn't support SSH1 based on Atsuhiko Yamanaka's comment) but svn is down right
> now so I can't commit it.

I have meant scp1 protocol, not ssh1 protocol.  Sorry for lack of words.
scp1 protocol is just a file transfer protocol and it has been used on ssh1 and ssh2 protocol.

For example, openssh'd sshd can disable ssh1 protocol support by configuring /etc/ssh/sshd_conifg,
but scp1 procol can be available if there exists '/usr/bin/scp' command.

Comment 24 J.M. (Martijn) Kruithof 2006-06-07 19:19:37 UTC
Stefan, have you upped the manual change?
Can this one be closed?
Comment 25 Stefan Bodewig 2006-07-27 04:37:59 UTC
See bug 39532 for a new sftp task patch, closing this one.
Comment 26 Atsuhiko Yamanaka 2006-09-27 09:35:12 UTC
Created attachment 18921 [details]
bugfix to download files/directory, which are pointed by symbolic link.

I have found a bug in downloading files and directories, which are pointed by
symbolic links,
with scp task using sftp of ant 1.7.0beta2.
I will attache a patch to fix this problem.
Comment 27 Stefan Bodewig 2008-07-14 06:44:45 UTC
*** Bug 39532 has been marked as a duplicate of this bug. ***