The more inundated we become with information, the greater the need for sorting it all out.
Sounding like something out of George Orwell’s dystopian novel 1984, the term ‘Big Data’ is widely used today, but is still sometimes a challenge to define.
According to the Merriam-Webster Dictionary, Big Data refers to “an accumulation of data that is too large and complex for processing by traditional database management tools.” Like many other terms, the meaning of Big Data has changed as technology has adapted, and our collective ability to gather, store, and disseminate information has grown.
The earliest known reference to ‘Big Data’ goes back to the time of the Second World War. Published in 1944, Fremont Rider’s book The Scholar and the Future of the Research Library: A Problem and Its Solution discussed the need to accommodate massive amounts of ‘data’ in the future, namely books.
Spending years researching the subject, Rider concluded that “no emendations in present library method alone were going to provide a sufficient solution of our growth problem,” and that every American college and university library had, “on the average, doubled in size every sixteen years” since they were founded.
Although Rider’s “growth problem” was later proved to be exaggerated, his argument was valid. With tens of thousands of books published annually in the United States at the time, where would they all be stored, and who would catalogue them?
Although other methods of information storage like microfilm and microfiche existed, they held many disadvantages, including the need for magnifying reader machines to make the small text legible, high cost, and difficulty in searching.
Adapting to technology
In the years since Rider coined the term, the meaning of Big Data has evolved. Between 1944 and the late 1990s, professors, authors, researchers, statisticians and others questioned not only how data was stored, but how it was accessed, absorbed, and comprehended.
Questions about data increased in the mid-1990s with the growth of the internet, as digital storage devices became more popular, less expensive, and able to contain vast amounts of information in smaller spaces than books.
In 2000, American economist and professor Francis X. Diebold presented a paper entitled ‘Big Data’ Dynamic Factor Models for Macroeconomic Measurement and Forecasting to the Eighth World Congress of the Econometric Society. In it, he wrote that science benefited from, and consequently had to find solutions for dealing with, Big Data, which he said “refers to the explosion in the quantity (and sometimes, quality) of available and potentially relevant data, largely the result of recent and unprecedented advancements in data recording and storage technology.”
From spreadsheet to software
While Big Data has existed for decades in rudimentary forms such as spreadsheets (columns and rows originally transcribed on paper long before software programs appropriated the name), it has expanded to an extent that few would ever have predicted.
Years ago, if we wanted to access ‘data’ in the form of a book, we went to our local library, opened labeled drawers in a stacked card catalogue, and flipped through thousands of typed (or even handwritten) index cards with title, author, and Dewey Decimal System numbers revealing where the shelved book could be found. While libraries still use the 144-year-old system, computers have hugely accelerated the process of finding books and information.
Extending beyond academics, Big Data today plays an important role in many industries. Global research and advisory business Gartner Inc. – which provides tools and advice for communications, marketing, finance, customer service and other sectors – has conducted studies on Big Data and organizations investing in the area.
One example is American multinational computer company Dell. The company naturally uses the most advanced methods to ease and improve the customer experience, and speed up customer support cases.
“It [Dell] is using big data analytics to analyze huge amounts of product and customer data gathered through the Internet of Things (IoT), which, in turn, provides Dell’s engineers with insights to resolve critical issues,” says the report summary.
“Big data has helped the company reduce issue resolution time by 84 percent, as well as prevent and eliminate faults before they occur, thanks to insights based on multi-customer data analytics.”
And Big Data is emphatically not confined to big business. Organizations or businesses do not require the scale of the 157,000 employee-strong Dell to find great value in embracing Big Data. Globally, countless small organizations with fewer than 100 employees are benefiting.
Within organizations, sales and human resources are two departments that in particular mine the information available from Big Data, frequently using it, for example, to reduce the time that a product takes to get to market.
Being able to amass and analyze more data, companies are able to focus their marketing efforts on specific potential customers, and conduct thorough analyses to ensure customers are satisfied with the product or service.
Energy to education
At work and in our homes, many of us already contribute information to Big Data without realizing it.
For instance, almost all places of employment have one thing in common – electricity. Necessary to power computers, keep the lights on, charge our phones, the same old electricity that used to be piped through old-fashioned electrical meters now comes to us through Smart Meters.
While metering the power we use, Smart Phones collect and store all the details, all the data, defining our power usage and then transmit this data wirelessly to the energy provider. By measuring how much energy is used and when, electrical utilities can charge customers by Time-of-Use.
And by tracking energy use online, clients can better manage their bills, sign-up for high-usage alert emails, review data on energy trends, and potentially save money.
Big Data plays an increasingly important role in other sectors, too, including education,. Although its adoption in the education sector has been slow – lagging behind technology, financial services, telecommunications and healthcare – Big Data is proving itself to be mutually beneficial for students and teachers.
For educators, Big Data can be used to continuously measure student performance, instead of the occasional standardized testing. Through adaptive learning methods, students can spend extra time on areas of difficulty while keeping-up with others in the class, and have progresses better monitored by teachers.
Additionally, teachers can have their performance monitored and measured, ensuring their lessons are on the right track.
Not to be outdone by education and energy, virtually all areas of our lives are being impacted by Big Data, including media, communications, entertainment, manufacturing, banking, healthcare, transportation, and government. It’s no surprise that many large entities use logistics and GPS to track our exact whereabouts, shopping and spending trends and habits, and a whole lot more.
While this relentless information-gathering can be annoying – witness the ads that pop up on Twitter, Amazon, and Facebook feeds – it is our ‘preferences’ that help customers, retailers, and financial institutions detect credit card theft and fraud. As hackers and scammers become more ingenious and brazen, Big Data plays a greater role in identifying unusual (read: ‘suspect’) purchases, and items bought from locations not in the card holder’s geographic area.
Likewise, insurance companies – who reckon that about 10 percent of their payouts already go on fraudulent claims – are using Big Data to minimize the collective billions of dollars covering these bogus claims. Collecting data in real time from numerous sources, including social media, insurance companies can flag strange patterns, for instance, someone who has been named as a witness in one or more earlier claims.
A business on the rise
Considering how many sectors are benefiting from Big Data, it comes as no surprise that investors are jumping aboard. According to reports from Research and Markets, the worldwide market for Big Data – valued in 2017 at $31.93 billion – is predicted to hit $156.72 billion by 2026, and keep growing. The reason is simple: the more industries adopting Big Data analytics to optimize their operations and collect information on customers, the bigger the demand.
From managing vast amounts of information to helping companies target their markets more precisely, to improving customer service, to helping students achieve better grades, and to fighting cybercrime, the world has come a long way from Fremont Rider’s “growth problem,” when managing information meant books and reports, not zillions of bits of digital data.
As long as Big Data keeps growing, our lives will be less private, but arguably made easier since nothing will ever be forgotten.