Federated Learning
For the past 30 or so years, technology has imposed a drastic change in everything. Including the human lifestyle. If we look back and see how things were 30 years back, it almost looks like everything about a person has changed in a way they can comply with the technical standards. Which results in the use of technology on a daily basis. This leaves us with a myriad of data generated worldwide. According to an article published in the World Economic Forum 2019, in collaboration with Visual Capitalist, by 2025, an estimation of 463 exabytes of data will be created each day globally.
The World Economic Forum article also states, “The exponential growth of data is undisputed, but the numbers behind this explosion — fuelled by the internet of things and the use of connected devices — are hard to comprehend, particularly when looked at in the context of one day”. This much data is a big deal because think about all the big things we can do with this data. We can analyze things, we can predict things, we can make simulated environments, we can give better user experiences. If we utilize this data in a meaningful manner, we can take the world to the next level of Artificial Intelligence. That’s why they say data is the new oil.
But, do you think we use this data at its fullest potential? Well, no! If so, what prevents us from doing that? As the World Economic Forum article suggests, most of this data lies in the edge devices. Like our smartphones. This introduces an element of privacy to this data which leads to legal issues when accessing or collecting these data and leaves us with more than enough resources to work with but with a dilemma in adhering to social and legal virtues.
What if, we can learn from everyone without learning about anyone? When I say learn from everyone, I mean learn from this big amount of data distributed all over the world. When I say without learning about anyone, I mean, do it while preserving privacy.
Well, that is where Federated Learning comes I handy. This is the definition for Federated Learning given by Peter Kairouz in the paper Advances and Open Problems in Federated Learning¹. After a thorough literature review, this is the best definition I found for Federated Learning.
Federated learning is a machine learning setting where multiple entities (clients) collaborate in solving a machine learning problem, under the coordination of a central server or service provider. Each client’s raw data is stored locally and not exchanged or transferred; instead, focused updates intended for immediate aggregation are used to achieve the learning objective.
Federated Learning was first talked about by Google in the years 2016/2017. And in 2017, their research team published Communication-Efficient Learning of Deep Networks from Decentralized Data², where they named their new distributed machine learning approach, Federated Learning. They say,
We term our approach Federated Learning since the learning task is solved by a loose federation of participating devices (which we refer to as clients) which are coordinated by a central server.
As the definition suggests, there are multiple clients. For example, smartphones. And there’s a central server that orchestrates the process. It initializes a global model and communicated it to all the clients. Clients can train the model using their data. The most interesting part is, data never leaves the client. Instead, the training is run in the client and they just send the training result to the server. The server aggregates all these trained models of all the clients to come up with a global model. The global model is broadcasted to the clients again. This cycle happens till the global model is good to go.
What just happened is Federated Learning. As simple as that! Isn’t it great! But, there are so many aspects to it. How does the server aggregate the locally trained models? What are the challenges faced while doing this? Have our smartphones been participating in this kind of training and we had no idea all this time? When did my smartphone participate in this process? What good has Federated Learning brought our day to day life? We can talk about all of that! In my coming set of articles. So stay tuned!
[1] Peter Kairouz, H. Brendan Mcmahan, Brendan Avent, Aur ́elien Bellet, and Mehdi Benniset al. Advances and open problems in federated learning. 2019.
[2] H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agueray Arcas. Communication-efficient learning of deep networks from decentralized data. InProceedings of the 20th International Conference on Artificial Intelligence and Statistics, pages 1273–1282, 2017.
