i'm using the following cmd:

C:\FDT\FDT-Windows-Client\FastDataTransferClient.exe -n Upload -c Disk -s Disk -d c:\Temp -M 1

Don't know what i'm doing wrong, never happen before in other machine. I changed the machine but i'm doing the same thing, what could be the problem?

I have this problem here:

Found 1 files in total
[10:49:00 INF] Transferring all files in source location
[10:49:00 INF] Adding up total size to transfer...
[10:49:00 INF] Total size to transfer is 0 MB
[10:49:00 INF] Session 1 of 1: Authenticated session successfully with id Pi21M
[10:49:00 ERR] Error in session initialization: FastDataTransfer.Exceptions.SessionInitializationException: Client elected to use
Disk Data Location on server, but that is not in the server's list of supported Location Types (which is Null, Storage)
at FastDataTransfer.AppStartup.ClientSideSessionFactory.AcceptEmptyAck(InboundBuffer buffer, BufferType expectedType) in D:_wo
rk\4\s\src\FastDataTransfer\AppStartup\ClientSideSessionFactory.cs:line 972
at FastDataTransfer.AppStartup.ClientSideSessionFactory.DoClientInitiatedSetup(TransferDirection direction, DataLocationType se
rverDataLocationType, ServerIdentificationParams serverParams, Int32 majorVersion, String authenticationKey, StorageParams storage
Params, BufferSenderParams bufferSenderParams, FileSpecParams fileSpecParams, FakeSourceParams fakeSourceParams, ILogger sessionLo
gger, SessionId& sessionId) in D:_work\4\s\src\FastDataTransfer\AppStartup\ClientSideSessionFactory.cs:line 866
at FastDataTransfer.AppStartup.ClientSideSessionFactory.CreateUploadSession(DataLocationType serverDataLocationType, Int32 majo
rVersion, ServerIdentificationParams serverParams, String authenticationKey, SizedFileList filesToTransfer, FileDatesOption fileDa
tesOption, IExtendedFileDestination fileSourceWithStats, StorageParams serverStorageParams, BufferSenderParams bufferSenderParams,
ReadInterceptCallback readInterceptor, WriteInterceptCallback writeInterceptor, ILogger sessionLogger) in D:_work\4\s\src\FastDa
taTransfer\AppStartup\ClientSideSessionFactory.cs:line 725
at FastDataTransfer.AppStartup.ClientSideSessionFactory.<>cDisplayClass18_1.<CreateInitializedSession>b3(<>fAnonymousType
22 <>h__TransparentIdentifier2) in D:\_work\4\s\src\FastDataTransfer\AppStartup\ClientSideSessionFactory.cs:line 564
at System.Linq.Enumerable.WhereSelectEnumerableIterator2.MoveNext()
at System.Linq.Enumerable.<CastIterator>d971.MoveNext()
at System.Collections.Generic.List1..ctor(IEnumerable1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source)
at FastDataTransfer.Sending.CompositeSendingSession..ctor(IEnumerable1 children, Int32 expectedTotalFileCount, Int64 expectedT
otalByteCount, ILogger logger) in D:\_work\4\s\src\FastDataTransfer\Sending\CompositeSendingSession.cs:line 35
at FastDataTransfer.AppStartup.ClientSideSessionFactory.CreateInitializedSession(TransferDirection direction, IEnumerable1 all
ServerParamsEnum, String authenticationKey, DataLocationType clientDataLocationType, String localDirectoryName, Nullable1 diskRea
dBufferSize, FileDatesOption fileDatesOption, StorageParams clientStorageParams, DataLocationType serverDataLocationType, StorageP
arams serverStorageParams, FileSpecBase fileSpec, FakeSourceParams fakeSourceParams, BufferSenderParams bufferSenderParams, Receiv
erCapacityParams receiverCapacityParams, IModDateFilter modDateFilter, IEnumerable1 alreadyCompletedFilesRelativeToRoot, ReadInte
rceptCallback readInterceptor, WriteInterceptCallback writeInterceptor) in D:_work\4\s\src\FastDataTransfer\AppStartup\ClientSide
SessionFactory.cs:line 578
at FastDataTransfer.AppStartup.ClientSideSessionFactory.RunOnce(IRawClientSettings parsedArgs, ILogger logger, ILogger fileConf
irmationLogger, ILogger fileErrorLogger, CancellationToken applicationCancellationToken, BufferSenderParams bufferSenderParams, Re
ceiverCapacityParams receiverCapacityParams, StorageParams serverStorageParams, TransferDirection direction, List1 allServerConne
ctionParams, StorageParams clientStorageParams, StorageParams senderStorageParams, IModDateFilter modDateFilter, IEnumerable1 alr
eadyCompletedFilesRelativeToRoot) in D:_work\4\s\src\FastDataTransfer\AppStartup\ClientSideSessionFactory.cs:line 279
[10:49:00 INF] Assessing whether retry is possible...
[10:49:00 WRN] Re-trying after failure. So far 0 files have been successfully transferred. Will attempt to transfer the remainder
by re-trying.
[10:49:00 WRN] Waiting 30 seconds before trying again because previous attempt transferred no files
[10:49:00 INF] If each attempt continues to transfer no files, at most 4 more retries will be made.

If you are transmitting across a WAN, there are a number of WAN Acceleration/Optimization products (Such as from Riverbed, Juniper, etc.) that can do this for you on the fly - some use Molecular Sequence Reduction algorithms to identify large chunks of data that have already been transmitted and merely sends a "token" to the other side of the WAN link, instead of redundantly transmitting that same data chunk redundantly.

In addition, some WAN acceleration/optimization algorithms perform ("spoof") the TCP three-way handshake interaction locally in order to reduce latency. This can have a greater effect than switching to a huge pipe (especially where the far - receiving - end cannot send the three-way TCP handshake responses fast enough.

Another way to improve effective throughput to employ a UDP-based network protocol (instead of a TCP-based protocol). UDP allows data packets to be sent back-to-back without an individual three-way handshake for each packet transmitted. It is also used to transmit latency-sensitive data, such as video or voice, but can be used for other purposes as well. For example, (TFTP Trivial File Transfer Protocol) is UDP-based. There can still be throttling/windowing mechanisms pushed back from the compute receiving that data packets (via transmitting messages on the same or a different port number or via a CONTROL message vs a DATA BEARING message). This method could also be used to indicate packets that never arrived (so that they can be retransmitted, for example). However, these retransmit and windowing requests could take place at the application layer, for example or somewhere in between that layer and layer 3 of the network stack.

A high-performance IP protocol, such as DDS, is an example of a very low latency protocol that uses UDP and can provide sub-millisecond response times across the same Ethernet switch. While DDS may not be oriented toward large file transfers, a two-way DDS session might be used as an out-of-band communication network to control rate of transfer, request retransmissions, etc.

If you have an Enterprise-grade Ethernet switch (such as from Cisco or Juniper), utilizing Jumbo packets can help eliminate the percentage of headers that make up the data stream to squeeze even more performance out of the system.

Another mechanism found in Enterprise-grade Ethernet switches is the network QoS settings that could prioritize certain network traffic (i.e.: the file copy protocol) over less-critical, less-time-sensitive data traffic.

If you want to send a SINGLE file image to MULTIPLE receivers, one area where the network switch MAY be able to provide some additional mechanisms to keep file transfer times to a minimum is to utilize IP MULTICAST. The beauty of IP MULTICAST is that ONE data stream can be effectively "split" into two or more data streams by the Layer 3 routing function within the Enterprise Ethernet Switch. This means that the SOURCE end of the file copying transaction would only send a SINGLE data stream with the file contents while MULTIPLE servers could receive their own individual copies of that exact same data stream. In Multicast, each receiver "subscribes" to that same multicast data stream that is being transmitted as ONE stream into the Layer 3 Enterprise-Grade switch. Assuming that the Multicast function of the data switch is capable of handling these extremely high-speed data streams, you would only transmit the file content ONCE to the switch and it would then distribute that stream to each receiving server that subscribed to that multicast. In the case of retransmit requests, they could be handled out-of-band via a high performance protocol, such as DDS, which could request the specific data block that was not received. This requires some programming to be performed that would coordinate the "out-of-band" DDS-based CONTROL messaging from the "BEARER" protocol that is used to transmit the file image data. Also, the software would need to take the steps to either sign up as the SOURCE of the multicast or as one of the DESTINATIONS of the multicast. Once initiated, a virtually unlimited number of DESTINATIONS could subscribe to the multicast and they would all receive a copy of the data stream SIMULTANEOUSLY (think of it as a limited broadcast).

NOTE: Even if the IP Multicast processing presented a performance slowdown, the overall transmission time should still be less than sending multiple unicasts one machine at a time. In the case of MULTICAST, we let the NETWORK perform part of the heavy lifting. Interestingly, that is how video data streams are delivered to thousands of endpoints simultaneously over large Enterprise networks - like the one where I work.

Also, you might want to consider creating a delta file PRIOR to transmitting to the destination servers. There are technologies such as those used by Harman RedBend MSM that pre-processes the before version and the updated version of the file to create a DELTA file. This technology was originally developed as a way to minimize network impact on cellular carrier networks when transmitting new firmware updates over the air to mobile devices.

One last technology that can be interesting is that which is used to keep remote SAN (Storage Area Network) clusters synchronized with each other. In those cases, only disc BLOCKS that have been altered since the last transmission are sent (another method of a differential update).

Anyway, it would be VERY interesting to experiment with a HYBRID approach using more than just one of these technologies to improve overall throughput of these file transfers in a complementary manner.

John Ruffing
Solution Architect
john_ruffing@hotmail.com