This chapter is mainly explained the data integrity review and management. This will be useful for data analytics team to ensure the data integrity to eliminate missing data or data anomaly to ensure the data is available and accurate for data analytics purposes. User also able to import or export the data for other usage.
3.1 Data Integrity Check
A data integrity check ensures that the data collected from devices is both available and accurate for intended business use cases. This section outlines common symptoms that may affect data integrity, along with the procedures to identify and validate any data inaccuracies that need to be addressed.
3.1.1 Data Concerns
3.1.1.1 Missing Footfall Data
| Descriptions | |
| Symptoms | No footfall in and/or out data is observed during the store opening hours for the entire day(s) or selected hour(s) |
| Pre-Requisites |
|
| Checking Procedures |
|
3.1.1.2 Missing Wi-Fi Counting Data
| Descriptions | |
| Symptoms |
No Wi-Fi counting data is observed for the entire day(s), which includes
|
| Pre-Requisites |
|
| Checking Procedures |
If the Pre-requisites have been fulfilled and yet the issue persists, raise a support ticket under "Data Accuracy Concerns" category to FootfallCam |
3.1.1.3 Suspected the Device is not Counting Accurately
| Descriptions | |
| Symptoms |
Site level footfall count is higher/lower than expected |
| Pre-Requisites |
* Repeat Step 1 and 2 for all the device(s) of the site which is intended to collect site level footfall data |
| Checking Procedures |
Two options for accuracy check:
If there is any data inaccuracy symptoms observed from the above accuracy checking processes, raise a support ticket under "Data Accuracy Concerns" category to FootfallCam with the video proof(s) or screenshot(s) |
3.1.2 How-to Guide
3.1.2.1 Check Device Status and Accuracy Audit Stage
Step 1 - Go to Settings > Device Page and check on Devices Health Check if the device is tagged with "Offline" tag. If yes, perform the device offline troubleshooting.
Step 2 - Click on Edit action link to go to the Device Settings page and check the following settings
-
Under Device Info tab, 'Active' is enabled
-
Under Settings tab, required counting usage(s) is enabled
-
Under Accuracy Audit tab, device is under '#3 Tuned' or '#4 Completed' stage
The following are the Accuracy Audit Stages for devices:
Noted:
- The data of a device is ready to be used only when its accuracy audit stage is either #3 Tuned or #4 Completed.
- As for the site data to be ready, all the active devices at the site must either be #3 Tuned or #4 Completed.
| Accuracy Audit Stage | Descriptions |
| #1 New | The device is newly allocated which requires usage and configuration confirmation. |
| #1 Ready to Tune | The device has been configured and is ready for tuning by FootfallCam. |
| #2 First Line Tuning | The tuning process of the device has been started and is expected to be completed within 3 working days under normal circumstances. |
| #2 Second Line Tuning | There is minor difficulty in tuning the device, requiring FootfallCam tuning specialist to look further into the issue. Please expect a minor delay in the accuracy audit completion date. (optional stage) |
| #2 Ticket Raised |
There is environmental constraints which restrain the accuracy audit process of the device from being able to be proceeded which requires action from client. Please refer to the relative support tickets for more info. (optional stage) |
| #3 Tuned | The device's settings have been refined and the data is ready to be used. In the meanwhile, FootfallCam will continue to collect sufficient sample size in order to prove the counting accuracy with video evidence. |
| #4 Completed | The device has successfully completed the accuracy audit with statistically significant sample size. Devices are expected to be completing accuracy audit within 7 working days under normal circumstances. Please click into the link to view the accuracy audit report. |
| No Need to Tune | Device's accuracy audit is put on hold due to testing purpose or constraints not being able to resolved, which would result in inaccurate counting for the device. |
| Reseller to Tune | The accuracy audit of the device will be carried out by FootfallCam Partner. |
3.1.2.2 How to remotely view Device Live View
The Live View is the current point-of-view of the camera. With the Live View, the user can:
1. Observe the live footage from the camera and adjust the placement of the camera if necessary.
2. View the placement of the In-Out line on the live footage and observe how the "In" and "Out" counts are triggered.
The feature of remotely viewing the Live View allows users to observe the footage on FootfallCam V9™ without needing to be in the vicinity of the camera. The live footage of counts being triggered also helps the user to instantly identify any factors affecting the accuracy of the counting, for example, a change in the environment covered by the camera that has significantly altered the path of human traffic.
If a user doesn't observe the Live View of his camera, he will not know if the camera's coverage and the In-Out Line are in the optimal configuration.
To remotely view the Live View, follow the steps below:
- Click "Settings" in the top navigation bar. Then, under "Settings" in the left navigation bar, click "Device". After that, navigate to the row of the device to observe and click "Edit" on that row.
- Use any one of the following methods:
- Method 1: Viewing in FootfallCam V9™
In the "Device Settings" page, click the "Calibration & Tuning" tab. Then, view the "Live View" tab.
-
- Method 2: Viewing via Setup Wizard
Step 1 - In the "Device Settings" page, click the "Settings" tab. Then, scroll down to the "Security & Privacy" section.
Step 2 - If the "Remote Access" selection is not set to "On", turn it on, click "Safe", and refresh the page.
Step 3 - Click on the link shown in the "Remote Access URL" field.
Step 4 - You'll be redirected to the device's Setup Wizard where you may log in. If you've forgotten your password, kindly contact "[email protected]".
Step 5 - You can now remotely observe this device's Live View.
3.1.2.3 Accuracy Audit
3.1.2.3.1 Download Accuracy Audit Report
This report allows users to verify footfall data accuracy, ensure data reliability, and support business decisions with 100% accuracy validation and video proof. There are two method to download the report:
Method 1 - Directly click on Download in the Device Management page.
Method 2 - Click on the View to be redirected to the corresponding accuracy report in Device Settings page (please refer Chapter 2: Device Management section 2.2.2.1.4).
After clicking Download or View to go the Accuracy Report page. The Summary and Validation Video will be displayed in this report. Users can scroll down to play the video or click Save as PDF to get a PDF report
3.1.2.3.2 Self Accuracy Auditing
User can perform the self accuracy auditing whenever the counting accuracy is in doubt by follow the step below:
Noted:
- Recorded videos will be temporarily stored for a maximum of 7 days.
- Verified vides will be temporarily stored for a maximum of 30 days.
- Create New Recording
Step 1 - Click on the Accuracy Audit in Device Settings page (refer Chapter 2: Device Management section 2.2.2.1.4) to open the Verifying Video page as below screenshot.
Step 2 - Click on the Create New Recording button at the top right.
Step 3 - Fill in all the necessary information
| Field | Description |
| 1. Site Operating Hour | Operating hour for the branch. (Info: For more information on Site Operating Hour, please refer to Section 4.5 Manage Site Detail.) |
| 2. Counter Local Date Time | Current date time for the device. |
| 3. Recording Type | Select the type of recording. (Choose 'Normal' for self auditing purpose.) |
| 4. Recording Date | Select the date to record. |
| 5. Recording Time | Select the time and duration to record. |
| 6. Add Schedule | Add another date to record |
| 7. Remove Schedule | Remove schedule date. |
- Perform Manual Counting
Noted: When the video status in Green is ready for verification purpose while Red is either pending for recording of the video is not uploaded to the server yet.
Step 1 - Go to Verifying Video page (refer Create New Recording)
Step 2 - Select any available video and click on Verify to perform manual count by referring the guideline as below:
|
|
|
|
| Visitor must go through the start-end of tracking zone (Red) and pass-by the In line (Green) then only will be counted as IN +1. | If two visitors go through the start-end of tracking zone (Red) at the same time and pass-by the Out line (Blue), both will be counted and OUT will be showing +2. | If two visitors go through the start-end of tracking zone (Red) at the same time and pass-by the In line (Green) or Out line (Blue), both will be counted as IN +1 and OUT +1 respectively. |
|
|
|
|
| Children (height < 1.3m) and inanimate objects will NOT be counted. | Visitor that make multiple U-turn within start-end of tracking zone (Red) and pass-by the In line (Green) will be counted as IN +1 only when he exits the tracking zone. | Visitor who go through the start-end of tracking zone (Red) and pass-by the In line (Green) but make a U-turn after exiting the tracking zone and pass-by the Out line (Blue) again will be counted as IN +1 and OUT +1. |
|
|
|
|
| Visitor (Head) that walk outside of start-end of tracking zone (Red) will NOT be counted. | Visitor that go through the start-end of tracking zone (Red) only without passing-by the In line (Green) or Out line (Blue) will NOT be counted. | Visitor that make U-turn within start-end of tracking zone (Red) will NOT be counted. |
Step 3 - Perform manual count with Keyboard Shortcut
Step 4 - Complete the process and click on Complete Verification Study button.
| Item | Description |
| 1. Video Recorded | The footage recorded to perform verification study. Red Line (Mandatory): Tracking zone. Blue Line (Mandatory): OUT line. Green Line (Mandatory): IN line. Yellow Line (Optional): Staff exclusion zone. |
| 2. Playback Speed | To adjust the playback speed of the video. |
| 3. Total System Count | To show the total visitor IN and visitor OUT counted in the video recorded. |
| 4. System Count | To show the live counting when playback through the video recorded. |
| 5. Manual Count | UP ARROW: Manual Count as OUT. DOWN ARROW: Manual Count as IN. |
| 6. Add Schedule | To show the accuracy of counting based on System Count vs. Manual Count. |
| 7. Remove Schedule | To show the manual count triggered with timestamp of the video recorded. |
- Delete Recording
Step 1 - Go to Verifying Video page (refer Create New Recording)
Step 2 - Check the box on the left for the video you wish to delete and click on Delete Video button at the top right, or click on the X of the particular row to delete the video
- Generate Accuracy Audit Report
Step 1 - Tick the video that want to include in the certified report/accuracy audit report. Also make sure the Video Status is green else the video in the report are unable to play.
Step 2 - Click on Generate Report to generate the report.
Step 3 - Fill in the information in the pop up and put name with FFC (FFC - name). Click on Generate to generate report.
Step 4 - Click on Yes to send the verification report to customer.
Step 5 - Fill in the related information and click on Send to send the report.
| Item | Description |
| 1. To | Enter Customer's Email. (Info: Type space or ';' to insert another email. Maximum of 5 email is accepted) |
| 2. Body | Enter the content of email. |
Noted: Make sure the video in the report are able to play or else customer will not able to play the video too. Regenerate the report is the video are unable to play.
Step 6 - The report will be generate successful, please refer Section 3.1.2.3.1 to view and download the accuracy audit report.
3.1.2.3.3 Live View/Video Audit for Accuracy Checking
Step 1 - Go to Settings > Device Page and click on Edit in Action column to go to the device settings page
Step 2 - Under Calibration and Tuning tab > Playback tab, click on Record Video Now button
Step 3 - Start Recording and leave comments when there is undercounting/overcounting is observed from the live view
Step 4 - End recording and save comments
(Note: Alternatively, you can also schedule video and perform accuracy audit when the video is available)
3.1.2.4 Data Audit in Data Integrity Workspace
Step 1 - Check the site data readiness,
If 'Site is Operating' and 'Site is Certified', proceed to Step 2
Else, device is not ready for accuracy checking yet
Step 2 - Check if there is any data anomaly symptoms can be observed from the 'Site Hourly Data for the Last 90 Days' and 'Site Hourly Data for the Last 7 Days' graphs
| Data Anomaly | Descriptions |
| Step Change |
A significant drop or increase in the data trend from a specific date onwards.  |
| Data Spike |
1. One-off Data Spike: Unexpectedly high counts occurring at a specific hour or day, happening only once.
Note: The following example is not considered as data spike as the difference between the highest value and other values is too small.
|
3.2 Data Preparation
Data preparation is a crucial process in transforming the raw data into a format that is suitable for analysis or modelling. In our system, we have ready with some data preparation job for our customer to perform the transformation before analysis.
| Job Type | Applicable Metric | Time Granularity | Entity Level |
| [Old & New Pipeline] Footfall Counting Data and Outside Traffic Data Patching |
|
|
|
| [New Pipeline] WiFi Data Patching |
|
|
|
| [Old & New Pipeline] Footfall Counting Data and Outside Traffic Data Patching |
|
|
|
3.3 Data Import and Export
As a data analytics system, user is able to import data from other source into our portal to obtain secondary metrics, such as import sales data to obtain the sales conversion and able to export our footfall data for other business purposes. Recurring data import export is available in the system, with pre-configuration.
To create import or export job, you may follow the steps below:
Step 1 - Login to Analytics Manager V9 portal via: https://v9.footfallcam.com/
Step 2 - Go to System > Import & Export to access the existing import & export job list.
Step 3 - To create new import or export job, click on the "+ Create a Job" button, then fill up the preferred job and select the job type, then click "Create" button to proceed.
| Item | Description |
| 1. Job Name | Enter your preferred job name. |
| 2. Job Type | Select the job type whether to import or export. |
Step 4 - You will be redirected to the detail import export job page according to the job type selected. Fill up all the details and click on the "Create Job" button to create the job.
Our system provides a lot of methods for data import and export, and the export files including different metrices. For more information, you may refer to the chapter below: