This chapter is mainly explained the data integrity review and management. This will be useful for data analytics team to ensure the data integrity to eliminate missing data or data anomaly to ensure the data is available and accurate for data analytics purposes. User also able to import or export the data for other usage.
3.1 Data Integrity Check
A data integrity check ensures that the data collected from devices is both available and accurate for intended business use cases. This section outlines common symptoms that may affect data integrity, along with the procedures to identify and validate any data inaccuracies that need to be addressed.
3.1.1 Data Concerns
3.1.1.1 Missing Footfall Data
| Descriptions | |
| Symptoms | No footfall in and/or out data is observed during the store opening hours for the entire day(s) or selected hour(s) |
| Pre-Requisites |
|
| Checking Procedures |
|
3.1.1.2 Missing Wi-Fi Counting Data
| Descriptions | |
| Symptoms |
No Wi-Fi counting data is observed for the entire day(s), which includes
|
| Pre-Requisites |
|
| Checking Procedures |
If the Pre-requisites have been fulfilled and yet the issue persists, raise a support ticket under "Data Accuracy Concerns" category to FootfallCam |
3.1.1.3 Suspected the Device is not Counting Accurately
| Descriptions | |
| Symptoms |
Site level footfall count is higher/lower than expected |
| Pre-Requisites |
* Repeat Step 1 and 2 for all the device(s) of the site which is intended to collect site level footfall data |
| Checking Procedures |
Two options for accuracy check:
If there is any data inaccuracy symptoms observed from the above accuracy checking processes, raise a support ticket under "Data Accuracy Concerns" category to FootfallCam with the video proof(s) or screenshot(s) |
3.1.2 How-to Guide
3.1.2.1 Check Device Status and Accuracy Audit Stage
Step 1 - Go to Settings > Device Page and check if the device is tagged with "Offline" tag. If yes, perform the device offline troubleshooting.
Step 2 - In the list view, click on "Edit" action link to go to the device settings page and check the following settings
-
Under 'Device Info' tab, 'Active' is enabled
-
Under 'Settings' tab, required counting usage(s) is enabled
-
Under 'Accuracy Audit' tab, device is under '#3 Tuned' or '#4 Certified' stage
3.1.2.2 Check Device Live View
Accessing to the device live view to check if the previously tuned parameter is still valid for current store environment or if the parameter has been changed since last tuned or certified.
Step 1 - Go to System > Data Accuracy Page and click on 'Live View' action link to go to the device settings page
Step 2 - The device live view will starts streaming and when the live view is available, see if the following is observed
-
Different store layout as compared to the previously tuned device live view (Next Action: Retuning is required)
-
Different tracking zone and/or in out lines in the current device live view as compared to the accuracy certificate (Next Action: Retuning is required)
3.1.2.3 Live View/Video Audit for Accuracy Checking
Step 1 - Go to Settings > Device Page and click on 'Edit' action link to go to the device settings page
Step 2 - Under 'Calibration and Tuning' tab > 'Playback' tab, click on 'Record Video Now' button
Step 3 - Start Recording and leave comments when there is undercounting/overcounting is observed from the live view
Step 4 - End recording and save comments
(Note: Alternatively, you can also schedule video and perform accuracy audit when the video is available)
3.1.2.4 Data Audit in Data Integrity Workspace
Step 1 - Check the site data readiness,
If 'Site is Operating' and 'Site is Certified', proceed to Step 2
Else, device is not ready for accuracy checking yet
Step 2 - Check if there is any data anomaly symptoms can be observed from the 'Site Hourly Data for the Last 90 Days' and 'Site Hourly Data for the Last 7 Days' graphs
| Data Anomaly | Descriptions |
| Step Change |
A significant drop or increase in the data trend from a specific date onwards.  |
| Data Spike |
1. One-off Data Spike: Unexpectedly high counts occurring at a specific hour or day, happening only once.
Note: The following example is not considered as data spike as the difference between the highest value and other values is too small.
|
3.2 Data Preparation
Data preparation is a crucial process in transforming the raw data into a format that is suitable for analysis or modelling. In our system, we have ready with some data preparation job for our customer to perform the transformation before analysis.
| Job Type | Applicable Metric | Time Granularity | Entity Level | Description |
| [New Pipeline] Footfall Counting Data Patching |
|
|
|
Data cleaning for missing values or inconsistencies. Value updated in hourly value and later reaggregated to daily value. |
| [Old Pipeline] WiFi Counting Data Patching (Hourly) |
|
|
|
Data cleaning for missing values or inconsistencies in WiFi Counting value. Hourly and Daily unique MAC will be different, no reaggregation required. |
| [Old Pipeline] WiFi Counting Data Patching (Daily) |
|
|
|
Data cleaning for missing values or inconsistencies in WiFi Counting value. |
| [Old Pipeline] Area In / Out Data Reaggregation |
|
|
|
Area footfall counting data reaggregation, to rerun the aggregation for hourly and daily, to correct late data issue. |
| [New Pipeline] In Out Data Correction for New Time Zone |
|
|
|
Corrective steps for wrong time zone set. |
| [New Pipeline] In Out Data Correction for New Operating Hour |
|
|
|
Corrective steps for wrong operating hour set. |
3.3 Data Import and Export
As a data analytics system, user is able to import data from other source into our portal to obtain secondary metrics, such as import sales data to obtain the sales conversion and able to export our footfall data for other business purposes. Recurring data import export is available in the system, with pre-configuration.
To create import or export job, you may follow the steps below:
Step 1 - Login to Analytics Manager V9 portal via: https://v9.footfallcam.com/
Step 2 - Go to System > Import & Export to access the existing import & export job list.
Step 3 - To create new import or export job, click on the "+ Create a Job" button, then fill up the preferred job and select the job type, then click "Create" button to proceed.
| Item | Description |
| 1. Job Name | Enter your preferred job name. |
| 2. Job Type | Select the job type whether to import or export. |
Step 4 - You will be redirected to the detail import export job page according to the job type selected. Fill up all the details and click on the "Create Job" button to create the job.
Our system provides a lot of methods for data import and export, and the export files including different metrices. For more information, you may refer to the chapter below: