Writing Custom Persistence Plugin

Currently, flytekit uses awscli or gcsutil by default to download the task’s input and upload the task’s output from the blob store. You may, however, want to write data to another storage service, such as HDFS or FTP. Flytekit provides a flexible way to extend and register a new persistence plugin.

Let us try to recap, why we should implement a persistence plugin

• Speed up blob store operations.

• Download the task input from a file system that Flyte doesn’t support.

• You have a custom authentication workflow in your own SDK

Add Custom Persistence Plugin

1. Extend DataPersistence and implement the interface.

2. Register it to the DataPersistencePlugins or overwrite the existing persistence plugin.

Example

Implement all the abstract functions in DataPersistence so that flytekit will know how to access and connect to your file systems.

class CustomPersistence(DataPersistence)::

    def __init__(self, name: str, default_prefix: typing.Optional[str] = None, **kwargs):
        self._name = name
        self._default_prefix = default_prefix

        @property
        def name(self) -> str:
            return self._name

        @property
        def default_prefix(self) -> typing.Optional[str]:
            return self._default_prefix

        def listdir(self, path: str, recursive: bool = False) -> typing.Generator[str, None, None]:
             # Returns true if the given path exists, else false
            raise UnsupportedPersistenceOp(f"Listing a directory is not supported by the persistence plugin {self.name}")

       def exists(self, path: str) -> bool:
            # Returns true if the given path exists, else false
            pass

       def get(self, from_path: str, to_path: str, recursive: bool = False):
            # Retrieves data from from_path and writes to the given to_path (to_path is locally accessible)
            pass

       def put(self, from_path: str, to_path: str, recursive: bool = False):
            # Stores data from from_path and writes to the given to_path (from_path is locally accessible)
            pass

       def construct_path(self, add_protocol: bool, add_prefix: bool, *paths: str) -> str:
            # if add_protocol is true then <protocol> is prefixed else
            # Constructs a path in the format <base><delim>*args
            # delim is dependent on the storage medium.
            # each of the args is joined with the delim
            pass

DataPersistencePlugins.register_plugin("cf://", CustomPersistence)

Here is also an example of adding a custom plugin by leveraging fsspec. Flytekit will use fsspec to do blob operation by default if the flytekit-data-fsspec plugin is installed.