A journey that began with just the human resource data going on cloud, to building full-fledged platforms (Media, Connected Car) integrated on cloud; the data migration journey has come a long way.
From on-premise big data / Hadoop implementation, to Spark jobs on cloud, and warehouses exclusively made for cloud, the journey for data has already begun. With the technological advancements in CSPs (Communications Service Providers)/ Hyperscalers and their capabilities to provide services under every umbrella (Infra, App migration, Data migration, ML implementations to Desktop-as-a-service), the focus now seems to be more on industry solutions.
The CSPs are now focusing on curating smart cities, connected car, media platforms, among others, for cloud. The market place of the CSPs is brewing more and more industry solutions with small boutique firms providing specialized niche offerings.
AI/ML (Artificial Intelligence/Machine Learning) models, which required heavy infrastructure and GPUs, found the serverless, virtually scalable infrastructure alluring and started migrating large datasets to be crunched on cloud. Seeing an opportunity here, CSPs and other third-parties also started providing datasets to cater to every industry. This further emboldened the CSPs to provide not just the infrastructure and runtime to run these ML models at production-scale but also the development ecosystem.
This barrage of data, data-crunching and data model training gave birth to many more cloud services in the form of DaaS (Data as A Service) and APIs (Application Programming Interface) to monetize data and share data. This entailed the creation of better network pipes to share/feed data and better options to migrate petabytes worth of data to cloud. This enforced the CSPs to focus on securing these networks and pipes and also create more secure virtual private networks to make systems work in a hybrid model, where on-prem and cloud systems worked hand-in-hand.
In the meantime, more and more niche products in the form of made-for-cloud ETL (Extract, Transform, Load) made-for-cloud warehouse, made-for-cloud database are coming up. Frameworks that were thought to be the only de-facto ways of implementing cloud migration journey, changed. ETL had to make way for ELT, with transformations, instead of being done inside the ETL tools where now being done on cloud, simply to leverage the scalability of cloud. Data that was crunched in an offline batch mode had to be revisited to work in asynchronous mode to ensure near-real-time (NRT) if not real-time feed of data to ML models.
Additionally, since some ML models have to be fed data in NRT, it became imperative that data quality and lineage was governed with more alacrity. DataOps became the new buzzword. Since data resides both on-prem and on cloud, a data virtualization implementation became imperative. Since the data was not just integrated with target systems but also fed in NRT to ML models, MLOps had to shake hands with DataOps.
The data on cloud ecosystem had been growing fast, with privacy becoming a critical factor, and with the likes of GDPR (General Data Protection Regulation) and CCPA (California Consumer Privacy Act) compliances becoming the norm. DataOps strategy had to be incorporated in the product lifecycle of every newly made product to ensure data could be securely stored and made available-on-demand if the need arose.
The whole new ecosystem that has emerged with the migration of data on cloud will further change once 5G becomes a reality. It will push the boundaries of what can be done with data on the edge and not just the cloud. That may start an individual fog/edge computing ecosystem war in itself.
Open Mobility foundation, which monitors the likes of smart city and connected car will bring in more providers and manufacturers together thereby bringing tech and industry further close together. This will further change the way devices or appliances are remotely monitored and repaired. Remote surgery/ Telesurgery is already a reality.
Cloud migration journey for data has truly come a long way and has even further accelerated in recent times. It is now transforming not only the tech but also humans in its own petabyte way!
