File Sharing Using External File Manager

Published Date: 02 Nov 2017

Abstract: People tend to have their data on more than one device these days. Security of this data is important along with its correctness. Cloud storage services are hence blooming in the market providing users with varied data storage facilities where client can store his data on the web and access it as and when required.

Having storage based service design in mind, we introduce an intermediate server to perform the job of authentication of client, confirming access rights, file download and upload. This intermediate server handles acceptance of credentials from the user, getting an authentication token and delivering it to the client so that the client can now directly access the object on the storage server. The metadata of the objects can be used for keeping the devices of a user in synchronization, later on.

Keywords: Object storage, intermediate server, file manager, file synchronization, cloud computing

Introduction

A large section of people use multiple computing devices these days. It is highly important to have this user data consistent on these devices. Storing the data on each device and then transferring the changes made on one to the next manually makes the user job highly cumbersome. So, people prefer outsourcing this task to cloud storage services.

Computation and storage resources are now a days provided by cloud computing. It also guarantees that your data is somewhere safe even in case of software failures, hardware failures or disasters at your side.

Object storage is one of the most fast maturing concepts today in industry. Both files and objects have metadata associated with the data they contain, but objects are characterized by their extended metadata. To have object storage, we have used Openstack Swift. Swift is a highly available, scalable, distributed, eventually consistent object/blob store. Swift allows to store a large amount of data efficiently, safely, and cheaply. For more information regarding Openstack Swift and its documentation, refer http://docs.openstack.org/developer/swift/ or the official Swift documentation at  http://swift.openstack.org.

In cloud storage, with the aim of reducing the data transfer latency, file sharing and storage solutions prefer direct client access to the storage server. This helps a great deal in removing the latency introduced by the intermediate server. However this goes with the bottleneck of having the intelligence of synchronization between the client side devices at the storage server.

This paper puts forth the approach of maintaining an intermediate server in between the client and the storage server. The procedure of authentication is carried with the help of this server itself. As we have the object storage at the server side, the intermediate serve can easily access the object metadata as well as store whatever of it is required in its database. This can also help it to handle the responsibility of synchronizing the user’s data on the different devices the user wants to be synced.

KEY TECHNOLOGIES

Object Storage

One type of storage new to most of the people today is object storage. Instead of providing file level of block level access to data, it provides access to whole objects or blobs of data. Different APIs are used for this job. Updation here takes place to the whole object.

The concepts used here are: (a) Account: one user has one account which is the way he accesses the storage. (b) Container: acts like a directory, it stores objects. (c) Object: a data file owned by a user, contained in a directory. One container can hold any number of objects.

Openstack Swift

The OpenStack Object Store project, known as Swift, offers cloud storage software so that you can store and retrieve lots of data in virtual containers.

Some of the quality features of Swift that prove to be very useful are as follows:

Scalability: Swift can store large amount of data in very cheap cost. It can scale horizontally so that there is no single point of failure. Swift can scale to thousands of machines with thousands of hard drives.

Redundancy: Each copy of object in Swift is stored in physically distinct zones. This prevents data loss and downtime due to common failures.

Unstructured data store: Swift does not store data as in a database. Data is stored here in bits. It thus stores blobs of data.

HTTP is used for any communication with Swift. Any action is requested using the HTTP verbs. Data access from web client is done here using the web service REST API. HTTP GET request downloads an object, HTTP DELETE request deletes an object, HTTP HEAD request fetches the object metadata, etc. The benefits of Swift that can be exploited here include its support to a high degree of concurrency, durability/availability, native REST-based HTTP API and comprehensive suite of language bindings.

Some of the benefits of Swift that can be exploited here are its support for high degree of concurrency, durability/ availability and native REST-based HTTP API with comprehensive suite of language bindings.

Openstack Object Storage provides:

Object Server that stores objects

Container server that keeps track of the of objects

Proxy server that handles all requests from the other server

Authorization server to authorize the cloud storage

Account server that keeps track of all the containers

All this can be combined for development or testing purpose on a single node which is called as Swift All In One (SAIO)

Swift Architecture

Proxy Server

For each request, the proxy server will look up the location of the account, container, or object in the ring and route the request accordingly. The public API is also exposed through the Proxy Server.

The Ring

A ring represents a mapping between the names of entities stored on disk and their physical location. When other components need to perform any operation on an object, container, or account, they need to interact with the appropriate ring of object, container or account to determine its location in the cluster. The Ring maintains this mapping using zones, devices, partitions, and replicas. Each partition in the ring is replicated, by default, 3 times across the cluster, and the locations for a partition are stored in the mapping maintained by the ring.

Object Server

The Object Server is a very simple blob storage server that can store, retrieve and delete objects stored on local devices. Each object is stored using a path derived from the object name’s hash and the operation’s timestamp. Last write always wins, and ensures that the latest object version will be served

Container Server

The Container Server’s primary job is to handle listings of objects

Account Server

The Account Server is very similar to the Container Server, excepting that it is responsible for listings of containers rather than objects.

Replication

Replication is designed to keep the system in a consistent state in the face of temporary error conditions like network outages or drive failures. The replication processes compare local data with each remote copy to ensure they all contain the latest version.

TempAuth

This is the Swift’s built-in authentication system. This is an excellent authentication module for Swift All In One (SAIO) for development/ testing work. The user of Swift passes in an auth token with each request. Swift validates each token. This token has an expiration time too, after which the user has to again authenticate himself. TempAuth also has the concept of admin and non-admin users within an account.

INTERMEDIATE (FILE-SYNC) SERVER

OpenStack Swift can be effectively used as a backend storage in cloud computing. However, people are becoming mobile today and hence use multiple computing devices. Getting the latest version of your data on all your devices no longer means emailing files back and forth or shuttling them on USB drives. Data consistency is very important to be maintained at all the ends here. Hence, file synchronization services based on storage cloud is becoming popular.

In our case, to meet the need of maintaining consistent data at user end, we use an intermediate server. Swift provides many effective ways to fetch the object metadata without accessing the object on the server. This fact is used by the intermediate server for the synchronization logic.

Prototype Implementation based on Intermediate Server (File sync server)

We have Openstack Swift object store at the storage server. Each client has one associated user account. In this prototype implementation, let us assume each user has one container in one user account and there is no account or container sharing. Our main objective is to achieve authentication and synchronization.

Authentication

The end user has the client application installed and configured against the Swift object storage and the file sync server, on all the devices he wishes to be synchronized.

Client has a particular directory which is synchronized.

The communication between the client and the file sync server takes place using a RESTful API.

The intermediate server has a database of user credentials and user device ids like MAC address.

In order to go ‘online’ on the application, the user has to use the installed command driven application to enter his credentials. The intermediate server then authenticates the user.

To perform further operations on the storage server, the client needs an authentication token which is obtained and stored by the intermediate server after using ‘curl’ utility on the object storage server (Swift). This token has a predefined expiration time and can be used for any operation that the client is authorized to do within that time. After expiration, client once again needs to authenticate itself.

If within the expiration time of a token, another device of the same user needs to access its data, it just has to authenticate itself by its credentials and the authentication token is shared. The intermediate server need not take another token for the new device.

Synchronization

In the case of users having multiple computing devices, we have proposed a synchronization method (which is yet to be implemented). The client directory has to be synchronized with the backend storage and the other directories on the other devices of the same user. The intermediate server can use the object metadata like file modification time for this logic. This is easily fetched from the object storage server without accessing the object. Client RESTful communication can be used to get such metadata from the client. This data will be processed to decide the synchronization direction and sequence. The synchronization operation will take place immediately when the client goes online or when the synchronizer runs periodically.

The periodic client inspection of the sync folder detects any changes made to it and notifies the same to intermediate (file-sync) server.

Upon signal from the intermediate server, client sends it the updated folder content along with the necessary metadata.

Intermediate server accordingly passes the same content to the storage and updates the user-object metadata database at its side at the same time.

The updated changes are passed down to other devices of the same user by their sync-folder inspection and synchronization process.

User operations on object storage

Once the client gets authenticated, GET, PUT operations have to be performed to read or write to its container.

Object Read: During the synchronization operation, the intermediate server issues the GET operation. The object then is fetched from the storage server and passed on to the client. The client directory thus remains updated.

Object Write: Whenever the client issues an upload instruction, the intermediate server accepts the file and uploads it to the storage device. Similarly, during uploading an edited object, the intermediate server PUTs the object on the object server. In case of users having multiple devices, the client will notify the intermediate server about the changes made to the directory. Accordingly, the intermediate server will run the synchronizer after a few checks and finally the synchronizer will sync the object appropriately.

Once the user goes offline, it can only access the locally stored objects in the directory. The changes made to the directory are synced only when the client goes online again.

Potential advantages of the proposed implementation

Since the communication among different entities is based on REST, it is easy to support various devices for file synchronization.

Since object storage has metadata capabilities, it helps intermediate server to use synchronization metadata.

Periodic synchronization process running at the client or the intermediate server is avoided as the client triggers the synchronization activity only when it detects changes made to its synced directory.

CONCLUSIONS AND FUTURE WORK

In this paper we have proposed an intermediate server based object storage for file storing and sharing. Extending metadata feature of object storage, authentication by intermediate server are few of the important implementation steps that are completed. Synchronization in case of multiple client devices is yet to be implemented. This implementation can be extended by allowing object or container sharing as well as allowing multiple containers in a single account. Also various conflicts ate yet to be handled that will arise during simultaneous updation, deletion or uploading of objects on different devices of same user.