Understanding big data is challenging because we, as a tech community, were not prepared for its sheer volume and the challenges it poses for storage, management, and analysis. About 90 percent of the world’s data has been generated in just the last two years, overwhelming us with information from various sources like social media, sensors, satellites, and more. Every day, over 2.5 quintillion bytes of new data are produced.
The difficulty arises not just from the quantity of data but also from the associated metadata, which is information about the data itself. Metadata includes details like the creation method, purpose, time, and location of data, and even the standards used. For instance, when you take a digital photo, the metadata includes information such as size, date, and location.
While metadata doesn’t make big data big, it does contribute to its complexity. To illustrate, understanding metadata for a photo involves looking at details like size, date, and location.
Now, let’s break down big data: it’s an immense amount of data, more than we’ve ever encountered, originating from diverse sources and accompanied by metadata. The challenge lies in handling, storing, and analyzing this vast and complex dataset.
When data becomes a problem purely due to its enormous size, it qualifies as big data. This is especially evident when dealing with data from various sources such as cell phones, satellites, sensors, and text messages. The complexity escalates when managing diverse data types and sources, as exemplified by a logistics company like UPS, which handles data from employees, trucks, and packages.
