Custom Azure Function FTP Extension

🗺️ Overview

We can use Azure WebJob SDK to develop a Azure Functions Custom Extension for FTP Triggers and Bindings.

1️⃣ Trigger

Triggers cause a function to run. A trigger defines how a function is invoked and a function must have exactly one trigger. Triggers have associated data, which is often provided as the payload of the function.

2️⃣ Binding

Binding to a function is a way of declaratively connecting another resource to the function; bindings may be connected as input bindings, output bindings, or both. Data from bindings is provided to the function as parameters.

• Input binding receives the data from the external source.
• Output binding sends the data to the external source.

More information about these two concepts can be found here.

📜 Requirements

The requirements for building this custom Azure Function FTP Extensions are:

We need a Ftp Triggers on:

• Single FtpFile or FtpStream
• Multiple FtpFiles or FtpStreams (= batch support)

And we need Ftp Bindings for:

• FtpFile
• FtpStream
• IFtpClient
• AsyncCollector

🏢 Building Blocks

Intro

In the next chapters I’ll highlight and explain all the components which are needed to build an FTP Azure Function Extension.

1. FTP Client

For connecting and listening to an FTP server a C# .NET FTPClient is needed, for this purpose I used FluentFTP:

FTP Trigger

To create a custom Trigger, we need to build these components:

2. TriggerAttribute

In an Azure Function, the triggers and bindings are defined by an Attribute class. This class holds all parameters and configuration values which are needed by the extension.

Define a class that extends from Attribute. This class represents our attribute class. We define all the parameters and configuration values for our trigger. In our case, we define the ConnectionString and properties.

1. FtpTriggerAttribute : This is the Attribute which the consumer should use to make the Azure Function trigger on an FTP event.
2. Connection (name or complete url) : The mandatory connection can be defined as a name (which should be defined in the ConnectionStrings section from a config file, or it can be the full connection string to connect to to the FTP server)
3. Configuration properties : Some properties are mandatory like the folder and polling interval. In addition there are also some optional properties.
4. The parameter : This is the parameter which receives the data when the trigger is triggered and sends data to this Azure Function. Note that only a predefined list of types are supported here, e.g. FtpFile, FtpFile[], FtpStream and FtpStream[].

First create an AbstractBaseFtpAttribute. This is an abstract base class because it’s used for trigger and binding attributes.

This class AbstractBaseFtpAttribute looks as follows:

public abstract class AbstractBaseFtpAttribute : Attribute
{
    /// <summary>
    /// The Connection represents the FTP connection string.
    /// </summary>
    public string Connection { get; set; } = null!;

    protected AbstractBaseFtpAttribute()
    {
    }

    protected AbstractBaseFtpAttribute(string connection)
    {
        Connection = connection;
    }

    /// <summary>
    /// Helper method to get ConnectionString from environment variable.
    /// If that fails, use the ConnectionString as-is.
    /// </summary>
    internal string GetConnectionString()
    {
        return Environment.GetEnvironmentVariable(Connection) ?? Connection;
    }
}

The FtpTriggerAttribute which extends the AbstractBaseFtpAttribute and looks as follows:

public sealed class FtpTriggerAttribute : AbstractBaseFtpAttribute
{
    /// <summary>
    /// The folder to listen on. Is optional.
    /// </summary>
    public string? Folder { get; set; }

    /// <summary>
    /// The maximum number of items returned in case an array is required.
    ///
    /// Default value is <c>32</c>.
    /// </summary>
    public int BatchSize { get; set; } = 32;

    /// <summary>
    /// The polling interval.
    ///
    /// Defined as {number}{s|m|h|d}.
    /// Examples:
    /// - 10s poll every 10 seconds
    /// - 2m  poll every 2 minutes
    ///
    /// Default value is <c>1m</c>. 
    /// </summary>
    public string? PollingInterval { get; set; }

    /// <summary>
    /// Gets files within subdirectories as well. Adds the -r option to the LIST command. Some servers may not support this feature.
    ///
    /// Default value is <c>false</c>.
    /// </summary>
    public bool Recursive { get; set; }

    /// <summary>
    /// Include the content from the FtpFile.
    ///
    /// Default value is <c>true</c>.
    /// </summary>
    public bool IncludeContent { get; set; } = true;

    /// <summary>
    /// Load the modify date using MDTM when it could not be parsed from the server listing.
    /// This only pertains to servers that do not implement the MLSD command.
    ///
    /// Default value is <c>true</c>.
    /// </summary>
    public bool LoadModifyDateUsingMDTM { get; set; } = true;

    /// <summary>
    /// Force a trigger when it runs for the first time, ignoring the modify date.
    ///
    /// Default value is <c>false</c>.
    /// </summary>
    public bool TriggerOnStartup { get; set; }

    /// <summary>
    /// The trigger mode to use.
    ///
    /// Default value is <c>ModifyDate</c>.
    /// </summary>
    public TriggerMode TriggerMode { get; set; } = TriggerMode.ModifyDate;

    public FtpTriggerAttribute()
    {
    }

    public FtpTriggerAttribute(string connection) : base(connection)
    {
    }
}

3. IListener

Define a class that implements the interface IListener.

The IListener interface has the following methods:

• StartAsync: – the Function Runtime calls this method to start our listener. This method returns one Task object that completes when our listener successfully started.
• StopAsync: – the Function Runtime calls this method to stop our listener. This method returns one Task object that completes when the listener completely stopped.
• Cancel: – the Function Runtime calls this method to cancel any ongoing listen operation.
• Dispose: – IDisposable’s dispose method.

The implementation class (FtpListener) from this interface contains the following logic:

• Connect to the FTP server
• FTP Listing
  • Recursive (gets files within subdirectories as well)
  • Detect changes since last run
• Get
  • FtpFile(s)
  • FtpStream(s)
• Process
  • Single FtpFile or FtpStream
  • Batch for FtpFiles or FtpStreams

The FtpListener class receives four parameters:

• ILogger; The logger
• Type; The Trigger value type (which can only be a FtpFile or FtpStream)
• ITriggeredFunctionExecutor; executor
• FtpTriggerContext; context

The FtpListener class uses the FtpClient (from FluentFTP) to listen to the FTP server. When data is received (in this implementation this is a FtpFile or FtpStream), we invoke the function using the ITriggeredFunctionExecutor instance.

For the full code from the FtpListener class, see this link.

4. ITriggeredFunctionExecutor

We use the ITriggeredFunctionExecutor instance to execute the triggered function when data (FtpFile or FtpStream) is received.

The FtpTriggerContext object has two member variables:

• The TriggerAttribute variable is an object of our Attribute class which used in the Azure Function and contains all the configuration (ConnectionString and other properties).
• The Client variable is an object of the IFtpClient interface which represents the FtpClient instance to connect FTP.

The context class looks as follows:

internal class FtpTriggerContext
{
    public FtpTriggerAttribute FtpTriggerAttribute { get; }
        
    public IFtpClient Client { get; }

    public FtpTriggerContext(FtpTriggerAttribute attribute, IFtpClient client)
    {
        FtpTriggerAttribute = attribute;
        Client = client;
    }
}

5. ITriggerBinding

Define a class that implements the interface ITriggerBinding. In this class, we create our listener and bind our trigger data. The ITriggerBinding-interface has the following methods:

• CreateListenerAsync: – the Function Runtime calls this method to create a listener. This method returns a Task object that has our listener.
• BindAsync: – This method is called to bind a specified value using a binding context. When our listener receives an event, we try to execute the function, passing the event data. This event data is encapsulated in a TriggeredFunctionData class and send to the Azure Function. In the BindAsync, we will bind this value to our corresponding data. This method returns a TriggerData class. The TriggerData class accepts a class that implements an IValueBinder interface and a read-only dictionary, this will revisited later in this article.
• ToParameterDescriptor: – the Function Runtime calls this method to get a description of the binding. This description is visible when running the Azure Function locally:
  
  And also when deployed in Azure:
  

6. ITriggerBindingProvider

Define a class that implements the interface ITriggerBindingProvider. This class is a provider class that returns a class that implements the ITriggerBinding interface. This class has the following methods:

• TryCreateAsync: – When functions are being discovered this method wil be called to get a class that implements the ITriggerBinding interface. the Function Runtime will pass a TriggerBindingProviderContext class as a parameter. In this method, we check whether the TriggerBindingProviderContext object contains our custom attribute and if the parameter has a valid type (e.g. FtpFile or FtpStream). If the Attribute is present and the parameter type is valid, a TriggerBinding class is created.

7. IValueBinder

Define a class that implements the interface IValueBinder. As explained in the BindAsync section, we are binding the trigger data to our data class using this class. The IValueBinder has three methods:

• GetValueAsync: – Returns a task that has the value object.
• SetValueAsync: – Returns a task that completes when the object to our data class completes.
• ToInvokeString: – Returns object as string. In case of a single FtpFile or FtpStream this is the FullName, in case of multiple objects (batch of FtpFiles or FtpStreams), this is a comma separated string from all the FullNames.

8. IExtensionConfigProvider

Create a class that implements the interface IExtensionConfigProvider. The IExtensionConfigProvider defines an interface enabling third party extension to register. This interface has the following method:

• Initialize: – in this method, all the triggers and bindings are registered, see code excerpt below:

  public void Initialize(ExtensionConfigContext context)
  {
      // 1. Add trigger
      var triggerRule = context.AddBindingRule<FtpTriggerAttribute>();
      triggerRule.BindToTrigger(new FtpTriggerBindingProvider(_logger, this));
  
      // 2. Add bindings
      var bindingRule = context.AddBindingRule<FtpAttribute>();
  
      // 2a. Add Input-Binding
      bindingRule.BindToInput<IFtpClient>(typeof(FtpBindingConverterForIFtpClient), this);
  
      // 2b. Add IAsyncCollector Output-Binding for FtpFile and FtpStream
      var arguments = new object[] { _logger, this };
      bindingRule.BindToCollector<FtpFile>(typeof(FtpBindingConverterForIAsyncCollector<>), arguments);
      bindingRule.BindToCollector<FtpStream>(typeof(FtpBindingConverterForIAsyncCollector<>), arguments);
  }
  

9. IWebJobStartup

And finally, we create a class that implements the interface IWebJobStartup. This interface defines the configuration actions to perform when the Function Host starts up. This interface has the following method:

• Configure: – the Function Runtime call this method when the function host initializes. in this method, we will add our custom extension.This startup class looks like this:

  public class FtpBinding
  {
      public class Startup : IWebJobsStartup
      {
          public void Configure(IWebJobsBuilder builder)
          {
              builder.AddFtp();
          }
      }
  }
  

Note the AddFtp method. This method is an extension function of IWebJobsBuilder and is defined in the FtpWebJobsBuilderExtensions class. The AddFtpExtension is a helper method. The FtpWebJobsBuilderExtensions class looks like this.

public static class FtpWebJobsBuilderExtensions
{
    public static IWebJobsBuilder AddFtp(this IWebJobsBuilder builder)
    {
        builder.AddExtension<FtpExtensionConfigProvider>();

        builder.Services.AddSingleton<IFtpClientFactory, FtpClientFactory>();
        builder.Services.AddLogging();

        return builder;
    }
}

As you can see in this extension method, we are adding the extension using the AddExtension method of the IWebJobsBuilder. The AddExtension method takes one parameter, our FtpExtensionConfigProvider instance. We are also adding a Singleton Service for the FtpClientFactory. The constructor of the FtpExtensionConfigProvider instance will receive this server as a parameter.

So basically what happens is when the Function Runtime starts, it searches for a class that implements IWebJobStartup. When it found a class that implements the interface:

• the Function Runtime calls the Configure method passing the IWebJobsBuilder object. We add the extension using the AddExtension method using the class that implements the IExtensionConfigProvider interface.
• the Function Runtime calls the Initialise method of the IExtensionConfigProvider passing ExtensionConfigContext as a parameter. Our implementation of the Initialize method adds the add the binding rule using the AddBindingRule method of the ExtensionConfigContext, which returns a BindingRule object. We call the BindToTrigger method to add our trigger passing TriggerBindingProvider as a parameter.
• After that system calls the TryCreateAsync function of the TriggerBindingProvider passing the TriggerBindingProviderContext as a parameter, in this TryCreateAsync method, we check whether our Attribute class present or not. A class that implements the ITriggerBinding interface is created and a Task that contains the object is returned.
• the Function Runtime then calls the CreateListenerAsync method of our class that implements the ITriggerBinding interface passing ListenerFactoryContext object. In our CreateListenerAsync, we return a class that implements the IListener interface. The ListenerFactoryContext object contains a class that implements the ITriggeredFunctionExecutor interface. The ITriggeredFunctionExecutor interface has a method called TryExecuteAsync. Using this method, we can execute the triggered function, passing the event data and CancellationToken.

FTP Binding

To create a custom Binding, we need to build these additional components:

10. BindingAttribute

Also for the binding, a class is required to represent our attribute class. We define all the parameters and configuration values for our binding. In our case, we define the ConnectionString and some extra properties.

1. FtpBindingAttribute : this is the Attribute which should be used by a consumer to indicate that this is an FTP Binding.
2. Connection (name or complete url) : This property is mandatory and can contain either the name of a corresponding ConnectionString in the config file, or it can be the full connection string to the FTP server.
3. Configuration properties : Some properties are mandatory like the folder. In addition there are also some optional properties.
4. The parameter : This is the parameter which receives the data when the binding is used in the Azure Function. Note that only a predefined list of types are supported here, e.g. FtpFile, FtpStream, AsyncCollector and IFtpClient.

Here is our FtpAttribute class (which extends the AbstractBaseFtpAttribute):

public sealed class FtpAttribute : AbstractBaseFtpAttribute
{
    /// <summary>
    /// The folder to listen for. Is optional.
    /// </summary>
    public string? Folder { get; set; }

    /// <summary>
    /// Call Connect() on the bound IFtpClient
    ///
    /// Default value is <c>false</c>.
    /// </summary>
    public bool AutoConnectFtpClient { get; set; }

    /// <summary>
    /// Cache the FtpClient based on the connection-string.
    /// In case of <c>false</c>, the caller must the dispose FtpClient manually.
    /// Default value is <c>true</c>.
    /// </summary>
    public bool CacheFtpClient { get; set; } = true;

    public FtpAttribute()
    {
    }

    public FtpAttribute(string connection) : base(connection)
    {
    }
}

Just like in the trigger, we have connection string (from the base-class) and a folder member variable.

8. IAsyncCollector

Create a class that extends the IAsyncCollector interface. This interface defines methods to AddAsync. The business logic can call the AddAsync function to send data to external resources (the FTP Server).

This class should handle FtpFile and for FtpStream and looks like:

internal class FtpAsyncCollector<T> : IAsyncCollector<T>
{
    private readonly ILogger _logger;

    /// <summary>
    /// FtpBindingContext instance
    /// </summary>
    private readonly FtpBindingContext _context;

    public FtpAsyncCollector(ILogger logger, FtpBindingContext context)
    {
        _logger = logger;
        _context = context;
    }

    /// <summary>
    /// UploadAsync to FTP
    /// </summary>
    /// <param name="item">Item to add</param>
    /// <param name="cancellationToken">A Cancellation Token</param>
    /// <returns>A Task that completes when the item is added.</returns>
    public Task AddAsync(T item, CancellationToken cancellationToken = default)
    {
        return item switch
        {
            FtpFile ftpFile => AddFtpFileAsync(ftpFile, cancellationToken),
            FtpStream ftpStream => AddFtpStreamAsync(ftpStream, cancellationToken),

            _ => throw new InvalidCastException($"Item of type '{item?.GetType()}' is not supported.")
        };
    }

    private async Task AddFtpFileAsync(FtpFile ftpFile, CancellationToken cancellationToken)
    {
        // Logic is defined here to upload a FtpFile using the IFtpCLient (via FtpBindingContext). ⭐
    }

    private async Task AddFtpStreamAsync(FtpStream ftpStream, CancellationToken cancellationToken)
    {
        // Logic is defined here to upload a FtpStream using the IFtpCLient (via FtpBindingContext). ⭐
    }
}

⭐ – For the full code from this class, see Bindings/FtpAsyncCollector.cs.

👁️ Class diagram overview

When all code is in place, you end up with the following class-diagram:

🧪 Testing

Testing the FTP Trigger

Now we need to create a sample function to test our trigger. Let’s create a test Azure Function that uses our trigger. Our sample function looks like this:

public static class FtpTriggerSample
{
    [FunctionName("RunFtpTriggerFtpFileAlways")]
    public static void RunFtpTriggerFtpFileAlways(
        [FtpTrigger("FtpConnectionAnonymous", Folder = "inbox", PollingInterval = "30s", TriggerMode = TriggerMode.Always)] FtpFile ftpFile,
        ILogger log)
    {
        log.LogInformation($"Info >> {ftpFile.Name} {ftpFile.FullName} {ftpFile.Size} {ftpFile.Content?.Length}");
    }

This method is straightforward, just log an informational message when this trigger is triggered by a new FtpFile on the FTP server. The ConnectionString is defined in the local.settings.json file, and the Folder is hard-coded as a string (‘inbox’).

Example local.settings.json configuration file:

{
    "IsEncrypted": false,
    "Values": {
        "AzureWebJobsStorage": "UseDevelopmentStorage=true",
        "FUNCTIONS_WORKER_RUNTIME": "dotnet",
        "FtpConnection": "ftp://testuser:mypwd@localhost",
        "FtpConnectionAnonymous": "ftp://localhost"
    }
}

Before running our function, we need to run an FTP server for testing (e.g. FileZilla).

For more examples, see Trigger.Sample.Ftp.

Testing the FTP Binding

Let’s create a sample function to test our binding. Our sample function looks like this:

public static class FtpBindingsSample
{
    [FunctionName("FtpBindingFtpFile")]
    public static IActionResult RunBindingFtpFile(
        [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)] HttpRequest req, // <-- The HttpTrigger request
        [Ftp(Connection = "FtpConnection", Folder = "inbox")] out FtpFile? item,          // <-- The FTP Binding on FtpFile
        ILogger log)
    {
        if (!req.Query.TryGetValue("message", out var stringValues))
        {
            item = default;
            return new OkObjectResult("Please provide a query parameter 'message' with a value.");
        }

        log.LogInformation($"Received message {stringValues}");

        item = new FtpFile
        {
            Name = "stef-ftpfile.txt",
            Content = Encoding.UTF8.GetBytes(stringValues.First())
        };

        return new OkObjectResult("FtpFile added");
    }
}

Note that this example function is triggered by a HttpTrigger and will get the message from the query and create a new FtpFile with name “stef-ftpfile.txt” on the FTP Server defined by the Connection “FtpConnection” (which is defined in the configuration file).

Unit-Testing the FTP Binding

The example below shows how to unit test a FTP Binding which uses an IAsyncCollector to add FTP Files to the FTP server.

Example Azure Function Code

The example Azure Function below is triggered by a HttpTrigger and will get the message from the query and append a new FtpFile with name “stef-asynccollector.txt” on the FTP Server using the AddAsync method from the IAsyncCollector<FtpFile> interface.

[FunctionName("BindingIAsyncCollector")]
public static async Task<IActionResult> RunBindingIAsyncCollectorAsync(
    [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)] HttpRequest req, // <-- The HttpTrigger request
    [Ftp("FtpConnection", Folder = "inbox")] IAsyncCollector<FtpFile> collector,      // <-- The FTP Binding on IAsyncCollector<FtpFile>
    ILogger log)
{
    if (!req.Query.TryGetValue("message", out var stringValues))
    {
        return new OkObjectResult("Please provide a query parameter 'message' with a value.");
    }

    log.LogInformation($"Received message {stringValues}");

    var item = new FtpFile
    {
        Name = "stef-asynccollector.txt",
        Content = Encoding.UTF8.GetBytes(stringValues.First())
    };

    await collector.AddAsync(item);

    return new OkObjectResult("FtpFile added to IAsyncCollector.");
}

Unit test code

The related unit-test looks as follows:

[Fact]
public async Task RunBindingIAsyncCollectorAsync_With_A_Message_Should_Call_AddAsync()
{
    // 1. Arrange : Request
    var value = "test";
    var bytes = Encoding.UTF8.GetBytes(value);
    var requestMock = new Mock<HttpRequest>();
    var queryParameters = new Dictionary<string, StringValues>
    {
        { "message", value }
    };
    requestMock.Setup(r => r.Query).Returns(new QueryCollection(queryParameters));

    // 2. Arrange : IAsyncCollector<FtpFile>
    var collectorMock = new Mock<IAsyncCollector<FtpFile>>();

    // 3. Act
    var response = await FtpBindingsSample.RunBindingIAsyncCollectorAsync
    (
        requestMock.Object,
        collectorMock.Object,
        Mock.Of<ILogger>()
    );

    // 4. Assert
    response.Should().BeOfType<OkObjectResult>().Which.Value.Should().Be("FtpFile added to IAsyncCollector.");

    // 5. Verify
    Expression<Func<FtpFile, bool>> match = f => f.Name == "stef-asynccollector.txt" &&
                                                 f.Content != null && bytes.SequenceEqual(f.Content);

    collectorMock.Verify(c => c.AddAsync(It.Is(match), It.IsAny<CancellationToken>()), Times.Once);
    collectorMock.VerifyNoOtherCalls();
}

1. Arrange the things needed to unit-test the Azure Function. So create a Mock for the HttpRequest and set the value of the query parameter “message” to “test”.
2. Create a Mock object from the <IAsyncCollector<FtpFile>, this Mock will be called with the AddAsync(FtpFile item) method.
3. Execute the method (RunBindingIAsyncCollectorAsync) on the System Under Test (FtpBindingsSample). Note that the requestMock and collectorMock are passed here. The logger is not tested here, so I just supply a default mocked object for ILogger.
4. Assert (check) if the return value is indeed a valid response.
5. Verify if the AddAsync method is indeed called with the correct parameter (= the FtpFile) and check if this FtpFile has the indeed the correct Name and byte-content.

💻 Software

• A NuGet package WebJobs.Extensions.Ftp can be downloaded here.
• And the GitHub project can be found here. In case you have questions or remarks, please create an issue there.

💡 Conclusion

This blog post describes step by step how to build a custom azure function extension (e.g. WebJobs.Extensions.Ftp). However when following the steps defined above, any external system can be handled by an Azure Function Trigger and Binding.

📺 Presentation

I did give a presentation (in Dutch) about ‘Azure Function Custom Extensions’ during the .NET Assemble! @ mStack, check out the recording below:

👏 Thanks

I want to thank Krishnaraj Varma for his excellent GitHub project NATS Custom Extension for Azure Functions which I used as a starting point to develop this WebJobs.Extensions.Ftp project and write this blogpost.

❔ Feedback or questions

Do you have any feedback or questions regarding my .NET Assemble! presentation or one of my blogs? Feel free to contact me!

📚 Additional resources

• https://github.com/robinrodricks/FluentFTP
• https://www.tpeczek.com/2018/11/azure-functions-20-extensibility_20.html
• https://geradegeldenhuys.net/read/custom-azure-function-trigger/