stuff to click

QFtp and FTPES: quick fix

difficulty 4 comments 3 added Feb 25, 2011 category Qt

For some time I was using my own CMS client to upload images to my sites. It was very convenient to use ready Qt solution — QFtp class. But after changing my hosting company I faced a trouble, new hosting accepted only secure FTP uploads. I searched the net looking for help on how to deal with FTPES with Qt but found just general recommendations to use QSslSocket class. So I explored the possibility of tweaking the QFtp class which also would minimize code changes in my app. This article demonstrates how to implement basic secure FTP with TLS authorization slightly tweaking QFtp. Though it is not the perfect solution it worked for me. The limitations of this approach is that it works only with explicit FTPS (FTPES), it doesn't work with certificates and also I sacrificed the active mode to keep things simple. But feel free to implement these yourselves!

What we have and what we need

FTP client makes a connection to server, sends credentials and establish data connection. Data connection uses another TCP socket and maybe established in two ways: passive and active. Passive mode is used when client can't accept connection on its listening port (like being behind the firewall) and PASV command is used to notify the server. Active mode is used when client can afford being TCP server for incoming data connection and PORT command is used to inform FTP server what port is open for data. Passive connection is simpler to implement because we don't have to deal with listening port, so from now let's cling to passive mode.

Normally QFtp connects to the listening socket of the FTP server and sends USER and PASS commands:

 
Response:	220---------- Welcome  ----------
Command:	USER egdfgdf
Response:	331 User egdfgdf OK. Password required
Command:	PASS ********
Response:	230-User egdfgdf has group access to:  egdfgdf  
Response:	230 OK. Current restricted directory is /
 

But we need to turn on encryption by sending AUTH TLS command before sending credentials:

 
Response:	220---------- Welcome  ----------
Command:	AUTH TLS
Response:	234 AUTH TLS OK.
Command:	USER egdfgdf
Response:	331 User egdfgdf OK. Password required
Command:	PASS ********
Response:	230-User egdfgdf has group access to:  egdfgdf  
Response:	230 OK. Current restricted directory is /
 

After logging in we prepare data transfer sending a list of commands, though some of them are optional (like SYST, FEAT), I left them to see details about the FTP server:

 
Command:	SYST //what system is running on server
Response:	215 UNIX Type: L8
Command:	FEAT //what features are supported
Response:	211-Extensions supported:
Response:	 EPRT
Response:	 IDLE
Response:	 MDTM
Response:	 SIZE
Response:	 REST STREAM
Response:	 MLST type*;size*;sizd*;modify*;UNIX.mode*;UNIX.uid*;UNIX.gid*;unique*;
Response:	 MLSD
Response:	 AUTH TLS
Response:	 PBSZ
Response:	 PROT
Response:	 ESTA
Response:	 PASV
Response:	 EPSV
Response:	 SPSV
Response:	 ESTP
Response:	211 End.
Command:	PBSZ 0 // Set protection buffer size, this goes before PROT
Response:	200 PBSZ=0
Command:	PROT P //Set protection level
Response:	200 Data protection level set to "private"
Command:	PWD
Response:	257 "/" is your current location
 

That's what we want to be implemented.

Brief QFtp anatomy

QFtp class is defined in /src/network/access/qftp.h file. It comes with other helper classes defined in cpp file:

  • QFtpDTP — Data Transfer Process, controlling client side data transfer
  • QFtpPI — Protocol Interpreter, dealing with FTP commands
  • QFtpCommand
  • QFtpPrivate

The main class QFtp defines enumerations for state, errors and commands, convenient interface (like cd(), get(), put()) and signals notifying about state changes. This is well documented in Qt help.

QFtpDTP class takes care of establishing data channel connection, reading and writing to the socket and parsing remote directory. The most important part for us is establishing connection, because we going to do encryption here.

QFtpPI class takes care of establishing command channel connection, sending commands and waiting replies. This class contains QFtpDTP object to make interactions with it simpler. Here we will need to implement additional commands, handle new replies and errors.

QFtpPrivate mainly contains essential data (QFtpPI object) and a set of private functions which helps running command sequence and handling errors.

Surgery

First of all let's copy qftp.h and qftp.cpp files to our working directory and rename the classes and files. I just make a batch replace of "QF" for "QZF" to get names like QZFtp. We are going to replace sockets based on QTcpSocket class with ones supporting encryption (QSslSocket) and enhance command set.

QFtp class tweaking

Basicly I would like to make very simple thing like just turning on TLS encryption. And at the QZFtp class level I expanded enumeration types:

 
//in QZFtp definition
    enum Error {
        NoError,
        UnknownError,
        HostNotFound,
        ConnectionRefused,
        NotConnected,
        SslError  //added
    };
 

and added one interface function:

 
//in QZFtp definition
void setTls(bool tls);
 

This function will set boolean variable depending on our plans to use encryption. The reasonable place for this variable is by all means QZFtpPI class, so we delegate variable setting to it:

 
void QFtp::setTls(bool tls)
{
    return d_func()->pi.setTls(tls); //don't forget to implement setTls in QZFtpPI !
}
 

Another function which need some work is QFtp::login(), as you remember, we need send "AUTH TLS" before sending credentials, and prepare encrypted data connection. New function will look like this:

 
int QFtp::login(const QString &user, const QString &password)
{
    QStringList cmds;
	if(d_func()->pi.isTls())
	{
		 cmds << (QLatin1String("AUTH TLS\r\n"));
	}
 
    cmds << (QLatin1String("USER ") + (user.isNull() ? QLatin1String("anonymous") : user) 
	+ QLatin1String("\r\n"));
    cmds << (QLatin1String("PASS ") 
	+ (password.isNull() ? QLatin1String("anonymous@") : password)
	+ QLatin1String("\r\n"));
 
	if(d_func()->pi.isTls())
	{
		 cmds << (QLatin1String("SYST\r\n"));
		 cmds << (QLatin1String("FEAT\r\n"));
		 cmds << (QLatin1String("PBSZ 0\r\n"));
		 cmds << (QLatin1String("PROT P\r\n"));
		 cmds << (QLatin1String("PWD\r\n"));
 
	}
 
    return d_func()->addCommand(new QFtpCommand(Login, cmds));
}
 

Here we assume that QZFtpPI has implemented isTls() getter function for our boolean variable.And the last thing we need to do with QFtp is comment out setTransferMode() function because we will work in passive mode.

In QFtp::put() function without encryption "STOR" command is preceded with "ALLO" command which is used to allocate enough space for file storing. In secure mode we don't need this:

 
int QFtp::put(const QByteArray &data, const QString &file, TransferType type)
{
...
     if(!d_func()->pi.isTls())	//added
          cmds << QLatin1String("ALLO ") + QString::number(data.size()) + QLatin1String("\r\n");
..
}
 

QFtpPI class tweaking

First of all, we remember to put these two functions in QZFtpPI definition:

 
	bool isTls() {return tls;}
	setTls(bool _tls) {tls=_tls;}
 

Also it would be nice to have signal void encrypted() for notifying that encryption is turned on and SSL errors notifier function:

 
signals:
    void connectState(int);
    void finished(const QString&);
    void error(int, const QString&);
    void rawFtpReply(int, const QString&);
    void encrypted();		//added
 
 
private slots:
    void hostFound();
    void connected();
    void connectionClosed();
    void delayedCloseFinished();
    void readyRead();
    void error(QAbstractSocket::SocketError);
    void sslErrors ( const QList<QSslError> & errors ) ;	//added
 

The last two things we have to do in QZFtpPI definition is to add boolean tls variable to private section of the class and change type of commandSocket variable to QSslSocket:

 
    QSslSocket commandSocket;	//was QTcpSocket
    QString replyText;
 
 
    char replyCode[3];
    State state;
    AbortState abortState;
    QStringList pendingCommands;
    QString currentCmd;
 
    bool waitForDtpToConnect;
    bool waitForDtpToClose;
    bool tls;		//added
    QByteArray bytesFromSocket;
 

Now let's look at the implementation, constructor is changed to:

 
QZFtpPI::QZFtpPI(QObject *parent) :
    QObject(parent),
    rawCommand(false),
    transferConnectionExtended(true),
    dtp(this),
    commandSocket(0),
    state(Begin), abortState(None),
    currentCmd(QString()),
    waitForDtpToConnect(false),
    waitForDtpToClose(false),
    tls(0)		//tls initialization added
{
    commandSocket.setObjectName(QLatin1String("QFtpPI_socket"));
    connect(&commandSocket, SIGNAL(hostFound()),
            SLOT(hostFound()));
    connect(&commandSocket, SIGNAL(connected()),
            SLOT(connected()));
    connect(&commandSocket, SIGNAL(encrypted()),
            SIGNAL(encrypted()));		// encrypted signal added
    connect(&commandSocket, SIGNAL(disconnected()),
            SLOT(connectionClosed()));
    connect(&commandSocket, SIGNAL(readyRead()),
            SLOT(readyRead()));
    connect(&commandSocket, SIGNAL(error(QAbstractSocket::SocketError)),
            SLOT(error(QAbstractSocket::SocketError)));		
	connect(&commandSocket, SIGNAL(sslErrors ( const QList<QSslError> & ) ), 	//sslErrors added
			SLOT(sslErrors ( const QList<QSslError> & ) ));	
 
    connect(&dtp, SIGNAL(connectState(int)),
             SLOT(dtpConnectState(int)));
 
	// additional settings for commandSocket variable
   commandSocket.setProtocol(QSsl::TlsV1); 		// we work with TLS
   commandSocket.setPeerVerifyMode(QSslSocket::VerifyNone); //we don't work with certificates
 
} 
 

Please note that in constructor we added some essential settings for command socket: now we are working in TLS mode and don't require certificate from the FTP server.

To send SSL errors with existing error() signal we are adding the following function:

 
void QFtpPI::sslErrors ( const QList<QSslError> & errors ) 
{
	QString e;
	for(int i=0; i< errors.size(); ++i)
	{
		e.append((errors[i].errorString())+"\n");
	}
 
	emit error((int)QFtp::SslError, e);
}
 

Now it is time to do some real things, after "AUTH TLS" is sent we should receive reply code 234 in case server supports it and doesn't need security data to proceed authentication (code 235). In first case we need to turn on encryption, in the latter we will send the error. This actions we will add to QFtpPI:processReply() function, since it is big enough I will show only the fragment that will be changed:

 
bool QZFtpPI::processReply()
{
...
else if (replyCode[0]==1 && currentCmd.startsWith(QLatin1String("STOR "))) {
		//dtp.waitForConnection();
        dtp.writeData();
    } 
	else if (replyCodeInt == 234 && tls) //TLS OK
	{
		commandSocket.startClientEncryption();
	}
	else if (replyCodeInt == 235 && tls) //TLS security data needed
	{
		state = Failure;
	}
...
}
 

Here we also commented out dtp.waitForConnection() call, because we don't need to wait data channel to be connected (passive mode). If other codes are received they will be processed in standard way, we only need this custom handling of 234 and 235 codes.

In bool QFtpPI::startNextCmd() it is OK to comment out processing of "PORT" command and replace it with debug notification, because this command can't be used in passive mode:

 
QZFtpPI::startNextCmd()
{
...
if (currentCmd.startsWith(QLatin1String("PORT"))) {
		qDebug(" Active mode - error!!! ");/*
        if ((address.protocol() == QTcpSocket::IPv6Protocol) &&
 transferConnectionExtended) {
            int port = dtp.setupListener(address);
            currentCmd = QLatin1String("EPRT |");
...
 }
 
        currentCmd += QLatin1String("\r\n"); */
 
    } else if (currentCmd.startsWith(QLatin1String("PASV"))) {
        if ((address.protocol() == QTcpSocket::IPv6Protocol) && transferConnectionExtended)
            currentCmd = QLatin1String("EPSV\r\n");
    }
...
}
 

QFtpDTP class tweaking

QFtpDTP class definition should be cleared from functions working in active mode, they are:

  • int setupListener(const QHostAddress &address);
  • void waitForConnection();
  • void setupSocket();

Also we need remove QTcpServer listener member and change QTcpSocket* type to QSslSocket* for socket member. That's all changes in class definition.

In implementation part we will start with constructor, where we remove all that relates to active mode:

 
QZFtpDTP::QZFtpDTP(QZFtpPI *p, QObject *parent) :
    QObject(parent),
    socket(0),
    //listener(this),
    pi(p),
    callWriteData(false)
{
    clearData();
//    listener.setObjectName(QLatin1String("QFtpDTP active state server"));
//    connect(&listener, SIGNAL(newConnection()), SLOT(setupSocket()));
}
 

And the last thing is to fix connection function:

 
void QFtpDTP::connectToHost(const QString & host, quint16 port)
{
    bytesFromSocket.clear();
 
    if (socket)
        delete socket;
    socket = new QSslSocket(this);	// was QTcpSocket
 
    socket->setObjectName(QLatin1String("QFtpDTP Passive state socket"));
    connect(socket, SIGNAL(connected()), SLOT(socketConnected()));
    connect(socket, SIGNAL(readyRead()), SLOT(socketReadyRead()));
    connect(socket, SIGNAL(error(QAbstractSocket::SocketError)), SLOT(socketError(QAbstractSocket::SocketError)));
    connect(socket, SIGNAL(disconnected()), SLOT(socketConnectionClosed()));
    connect(socket, SIGNAL(bytesWritten(qint64)), SLOT(socketBytesWritten(qint64)));
 
    socket->connectToHost(host, port);
	if(pi->isTls())	// here we need to setup QSslSocket as we did in QZFtpPI class
	{
		socket->setProtocol(QSsl::TlsV1);		// TLS mode
		socket->setPeerVerifyMode(QSslSocket::VerifyNone);	// no certificates
		socket->startClientEncryption();			// starting encryption
	}
 
}
 

Putting it together

In qzftp.h file place string

#include "qglobal.h"

before other include directive. Also we don't need #ifndef QT_NO_FTP and Q_NETWORK_EXPORTmacros in this file. In qzftp.cpp don't forget to include qzftp.h instead of qftp.h and qsslsocket.h instead of qtcpsocket.h !

As you could see in the end of qzftp.cpp there are two other include directives :

 
#include "qzftp.moc"
 
#include "moc_qzftp.cpp"
 

If you try to compile this cpp file before mocking it, you will get an error "Cannot open include file: 'qzftp.moc'". To mock it just build your project.

Download tweaked sources!

How to use it

This is the simplest part. If you know how to use QFtp it is pretty much the same, you only need to call setTls() member function before connecting:

 
...
ftp = new QZFtp();
ftp->setTls(bUsingTls); //add this
 
connect(ftp, SIGNAL(commandStarted(int)), this, SLOT(OnFtpCommandStarted(int)));
connect(ftp, SIGNAL(commandFinished(int,bool)), this, SLOT(OnFtpCommandFinished(int,bool)));
connect(ftp, SIGNAL(done(bool)), this, SLOT(OnFtpDone(bool)));
connect(ftp, SIGNAL(dataTransferProgress(qint64,qint64)), 
	this, SIGNAL(DataTransferProgress(qint64,qint64)));
 
ftp->connectToHost(host);
...
 
 

Comments

fan: SFTP?

added Mon, 28 Jan 2013 09:04:22 -0500
is SFTP possible with this also?
tried this, but cannot login ...

why is there no basic SFTP Support in Qt ... :(

fan: thanks at all

added Mon, 28 Jan 2013 09:05:23 -0500
... but big thanks for your detailed work!

Bob ONeil: 3 Issues

added Fri, 25 Apr 2014 12:10:39 -0400
First, many thanks for this effort, it accomplishes the exact goals I have. I have tweaked your source a bit to allow for your tweaked source to build as a standalone class within a Qt source application. I am having 3 problems with solution, that I wondered if you had an interest in resolving to make the class even better. I am testing using the vsftpd server under Linux, and use standard browsers to verify FTP client behavior in non-secure mode to view the messaging.

1. FTPS uploads: the upload is incomplete on the vsftpd server, it is always missing the last block of the upload. The tracing on the client side between the FTP and FTPS uploads seems identical, so the client side seems happy. On the vsftpd side, it indicates the reception of the truncated file size prior to a connection termination. I wonder if the QUIT is being issued prior to SSL decode.

2. FTPS Download: if no prior connection is made using FTP, the FTPS routine seems to run properly to the download phase, but then the file is never closed on the FTP client, QUIT is never issued. There is no problem doing an FTP download.

3. FTP followed by FTPS: when an FTP download occurs prior to an FTPS session, when the latter is kicked off, the command sequence is such that the transfers never occur. It is almost as if some state information (such as connected) is being maintained.

Please email me at "boneil@cinci.rr.com" for the attachments.
Place a comment