Data Collection Methods and Data Mining Techniques 08 October, 2020   

Data is the information used to prove a point or substantiate a decision.  The quality of the decisions we make is dependent on the quality of data that backs it. Data mining relates to finding knowledge from data collected. The data collection and data mining tools are central to most corporate, administrative, and personal decisions. 

Data are of two types: 

Primary data are those collected to make a particular decision with a pre-defined objective. Primary data is defined as the data collected by the researcher or funded by him/agency with a specific object in view. Generally, primary data maybe for a small location, short duration, and a specific audience. The best example is data collected by organizations from their customers.

Primary data collection can be time-consuming, resource-intensive, and expensive, depending on the size, purpose, and duration.

Secondary data is collected for a related purpose by some other agency or researcher. Hence usability, reliability, and aim for which data is collected or the source can affect the quality of the decision. But in some instances, only secondary data will be the available source. 

Caution on Secondary data–

• Shall be used only if primary data is not available
• In almost all research where secondary data is the only method of obtaining information, it is collected from reliable, published, and attributed sources. This is very important as research validity and reliability is based on the source of data.

The authenticity of the source has to be checked:

• Financial market research data on the stock price movements can be conducted only with secondary data from the relevant stock market. Data collected from a company's website or its audited balance sheet of a corporation is reliable data. Data released by ACC and ACCA are considered reliable. Data collected from trade journals are also considered authentic. 

Data reliability is important:

• Data released by the Federal Government or any Government agency can be considered reliable. Data that is independently verifiable is deemed to be reliable. Data collected in another country's location with the same purpose/questionnaire can be considered safe for comparison.

Population: Whatever data collection methods, one needs to understand the word's population and sample. The population is the actual number of respondents or the relevant people for the study.

For example: To find the buying behavior of Mexican meat-eaters in California, USA, the relevant population is the total meat-eaters of CA who have migrated from Mexico and not the entire population of meat-eaters of CA.

Qualitative and Quantitative Data

Quantitative data is in numbers, whereas qualitative data is about feeling emotions and ranking of the subject of analysis. For all qualitative data research, one will have to use only primary data as secondary data can only substantiate conclusions and lack reliability. 

Data Collection Methods of Primary Data:

• Sample Survey: In general, individual researchers and small organizations have limited time and resources. They shall adopt a limited scope census for collecting data, and the results need to be checked for representativeness. Sample-based surveys reduce the number of individuals to be enumerated. For that, one needs to find a logical way to decide the sample size for the survey. 

• Trade patterns based on tax collected is a reliable source of primary data. Data collected from trade associations regarding sales trends are relevant primary sources of data. 

• Logbook Entries: To find the number of people traveling in a day by cars between two destinations, one can use logbook entries at toll booths.

• The process of Data Collection: Once a researcher or an organization decides on the purpose of data collection, they proceed to decide the nature of data collected: primary or secondary; Method to be used: census or sample; and then the next step is the technique adopted to collect data

The process of data collection: 

There are many techniques used to collect data. Some of them are: 

• Interviews: They can be of many types; 

1. Personal

2. Telephonic

3. Direct personal investigation

4. Indirect oral investigation

5. Structured interviews

6. Unstructured interviews 

7. Focused interviews

8. Clinical examinations

9. Non-directive interview

• Focus groups

• Questionnaires

• Schedules

• Observation 

These are the traditional methods of collecting data. These days, qualitative data are collected using computers and CCTV’s to understand human behavior in different situations rather than through manual methods.

After the data collection methods are complete, the first step is to filter the available data to remove errors and irrelevant or odd information. After that, the data is warehoused, and only relevant data is moved over to data mining. Warehoused data can be reused to create relevant insights in the future. 

Data Mining:

Data mining is a computer science-related process by which the data collected is useful for understanding and decision making. It is also called the discovery of knowledge in databases. It is the process of finding related information from the collected data. It involves classifying the collected data, organizing it into a useable form, and discovering patterns. It uses statistics and database systems for analyzing and understanding. Although data mining is beneficial and enables better decision-making, it involves compromising the security, safety, and privacy of the people involved.

What new did we get from Data mining?

We could always collect a lot of data. Statistics helped classify, organize, and analyze in simpler ways to find associations and relationships at a smaller scale. But data mining can use machine learning and analytics to process large data to extract various patterns, relationships, and combinations. It enables seemingly trivial information to draw useful, effective knowledge that can help make meaningful decisions. Even a small business can use it to their advantage. As data analytics tools and techniques have improved in multiple proportions, data mining has become highly efficient and gives quick results. Data mining has moved from interpreting data to untangling data into tiny bits to draw conclusions at micro-levels.

Data mining helps in every process of business: 

The most basic economic question of ‘What to produce, how to produce, and for whom to produce’ can be answered by data mining.

• Better procuring of materials at most suitable prices and materials management. 

• Improved efficiency of operation through operational research techniques

• Storage, management, and logistics of inventory

• Marketing and selling efficiency by better placement of goods at stores, logistics, and e-commerce

• Aftersales service by faster and efficient coordination of resources at economical rates.

Over and above, these data mining supports business in target marketing, market segmentation, cross-selling, and Customer Relationship Management.

We can achieve customer retention, customer profiling, efficient forecasting, effective quality control, and competition analysis by monitoring competition and pricing strategy for every product.

Techniques of Data mining: 

• Association, to find a relationship between facts available

• Classification to arrange data in a clear form 

• Clustering, making data into clusters from different areas for comparison.

• Prediction is to extrapolate from various perspectives and drawing results.

• Sequential patterns are found from large data to help find sequences in data. 

• Decision trees

• Combinations

• Long-term memory processing

Hence data becomes redundant without data mining, and data mining cannot be carried out with junk data. The data mining process can be custom-made to meet the business requirements, be it large or small. Whether its small, medium, or large-scale businesses, everyone needs to adopt proper data collection methods to propel their business forward. Only the scope of collection changes with the size of the organization.