.jpg)
The blockchain approach to data storage problems
Sunday, May 13, 2018
|
Pavel Bains |
How decentralized blockchain storage of data could be the answer to the growing demands information is putting on the modern cloud data centers.
The human race has truly entered the Information Age. Once the epitome of innovation, the floppy disk’s performance is now eclipsed by thumbnail-sized SD cards that boast over 500,000x the capacity of the already-ancient technology. Terabyte-denominated drives are becoming the norm, and prefixes like peta-, exa- and zetta- are being used to quantify the mind-boggling amounts of data flowing in cyberspace.
Far from being a buzzword, the term Big Data refers to the processing of exponentially growing data sets. As advances in the Internet of Things and Artificial Intelligence power forward, the strain on storage is only set to worsen. New data centers are cropping up at a rapid rate to alleviate this strain, but will it be enough?
IDC predicts that, by 2025, there will be 163 zettabytes of data in the world. An IBM report, in the same vein, found that 80% of data today is unstructured - that is, text-heavy information that cannot be properly interpreted using traditional analytical methods. Companies collect hordes of this data, but, unless they can derive insights from it, it serves little purpose in growing their business. Instead, they begin to stockpile the unstructured data, incurring costs as they must constantly upgrade their storage capacity to make room for the useless information.
Cloud databases, in their current iteration, leave much to be desired by way of scalability - expanding capacity under such a model proves to be costly. For dApps (or ‘decentralised applications’) bolted on top of blockchains, there is a need for a more purpose-built method of storage to handle the titanic amounts of data these applications consume. Blockchains by themselves, whilst technically databases, are not built for high-volume use-cases.
There is a pressing need for a new solution that reflects the philosophical underpinnings of a decentralised web. Notably, one that embraces the power of a distributed infrastructure to not only scale efficiently, but to harness the security afforded by blockchain technology to avoid data-breaches and eradicate the all-too-prevalent single point of failure architecture.
Decentralisation in data storage has already proven to be highly effective in reducing costs for the end user - instead of relying on a few companies with dedicated data centers (offering pricing that reflects a relatively uncompetitive market), blockchain protocols have enabled users to monetise unused storage space (or indeed, rent out dedicated rigs), with few barriers to entry. The creation of a free market for storage not only vastly lowers costs as hosts compete against each other, but also gives purpose to the troves of otherwise unused space scattered across the globe.
Another edge the blockchain approach has over traditional cloud storage is that of dependability: disasters in geographical locations that encompass central databases can range from minor nuisances caused by power outages, to severe ramifications halting business operations as an earthquake destroys one such centre. This problem is nonexistent in distributed networks, as redundant copies of data are dispersed across a number of nodes. In the case where one node goes offline (whether temporarily or permanently), an identical replica of the data can simply be retrieved from another - as such, a user has access to their data around the clock, with no downtime.
Immutable and secure storage is impressive by itself. For real-time applications in IoT and related fields, however, speed is of vital importance given the constantly flowing data from device to device. The concept of swarming is proposed as a method for ensuring minimal latency to guarantee high-speed transfers. Much like torrenting, devices within a swarm - a large group of nodes in a peer-to-peer network - will retrieve data from the nearest and fastest nodes (querying files by the hash of their content). Leveraging a platform such as the Ethereum blockchain, however, adds an incentivisation layer to the traditional approach seen in torrenting - nodes are rewarded for replicating data.
It would be overly optimistic to say that the technology is already perfected. Blockchain tech is still in its infancy, and there are still elements in need of optimisation (two that spring to mind are scaling the blockchains themselves and reducing transaction fees). Once the kinks have been ironed out, there is then what is perhaps the largest barrier to overcome - adoption. That said, when blockchain platforms have proven themselves to be effective, the offering of better security, reliability and lower costs will no doubt see businesses flock to distributed databases for their storage needs.
This content is made possible by a guest author, or sponsor; it is not written by and does not necessarily reflect the views of App Developer Magazine's editorial staff.
Far from being a buzzword, the term Big Data refers to the processing of exponentially growing data sets. As advances in the Internet of Things and Artificial Intelligence power forward, the strain on storage is only set to worsen. New data centers are cropping up at a rapid rate to alleviate this strain, but will it be enough?
IDC predicts that, by 2025, there will be 163 zettabytes of data in the world. An IBM report, in the same vein, found that 80% of data today is unstructured - that is, text-heavy information that cannot be properly interpreted using traditional analytical methods. Companies collect hordes of this data, but, unless they can derive insights from it, it serves little purpose in growing their business. Instead, they begin to stockpile the unstructured data, incurring costs as they must constantly upgrade their storage capacity to make room for the useless information.
Cloud databases, in their current iteration, leave much to be desired by way of scalability - expanding capacity under such a model proves to be costly. For dApps (or ‘decentralised applications’) bolted on top of blockchains, there is a need for a more purpose-built method of storage to handle the titanic amounts of data these applications consume. Blockchains by themselves, whilst technically databases, are not built for high-volume use-cases.
There is a pressing need for a new solution that reflects the philosophical underpinnings of a decentralised web. Notably, one that embraces the power of a distributed infrastructure to not only scale efficiently, but to harness the security afforded by blockchain technology to avoid data-breaches and eradicate the all-too-prevalent single point of failure architecture.
Decentralisation in data storage has already proven to be highly effective in reducing costs for the end user - instead of relying on a few companies with dedicated data centers (offering pricing that reflects a relatively uncompetitive market), blockchain protocols have enabled users to monetise unused storage space (or indeed, rent out dedicated rigs), with few barriers to entry. The creation of a free market for storage not only vastly lowers costs as hosts compete against each other, but also gives purpose to the troves of otherwise unused space scattered across the globe.
Another edge the blockchain approach has over traditional cloud storage is that of dependability: disasters in geographical locations that encompass central databases can range from minor nuisances caused by power outages, to severe ramifications halting business operations as an earthquake destroys one such centre. This problem is nonexistent in distributed networks, as redundant copies of data are dispersed across a number of nodes. In the case where one node goes offline (whether temporarily or permanently), an identical replica of the data can simply be retrieved from another - as such, a user has access to their data around the clock, with no downtime.
Immutable and secure storage is impressive by itself. For real-time applications in IoT and related fields, however, speed is of vital importance given the constantly flowing data from device to device. The concept of swarming is proposed as a method for ensuring minimal latency to guarantee high-speed transfers. Much like torrenting, devices within a swarm - a large group of nodes in a peer-to-peer network - will retrieve data from the nearest and fastest nodes (querying files by the hash of their content). Leveraging a platform such as the Ethereum blockchain, however, adds an incentivisation layer to the traditional approach seen in torrenting - nodes are rewarded for replicating data.
It would be overly optimistic to say that the technology is already perfected. Blockchain tech is still in its infancy, and there are still elements in need of optimisation (two that spring to mind are scaling the blockchains themselves and reducing transaction fees). Once the kinks have been ironed out, there is then what is perhaps the largest barrier to overcome - adoption. That said, when blockchain platforms have proven themselves to be effective, the offering of better security, reliability and lower costs will no doubt see businesses flock to distributed databases for their storage needs.
This content is made possible by a guest author, or sponsor; it is not written by and does not necessarily reflect the views of App Developer Magazine's editorial staff.
Become a subscriber of App Developer Magazine for just $5.99 a month and take advantage of all these perks.
MEMBERS GET ACCESS TO
- - Exclusive content from leaders in the industry
- - Q&A articles from industry leaders
- - Tips and tricks from the most successful developers weekly
- - Monthly issues, including all 90+ back-issues since 2012
- - Event discounts and early-bird signups
- - Gain insight from top achievers in the app store
- - Learn what tools to use, what SDK's to use, and more
Subscribe here
