Table of Contents
At a glance
Description
The analyzeMissingPatterns action performs a missing pattern analysis. It is a part of the Data Science Pilot action set, designed to automate and enhance data science workflows. This action is particularly useful in the exploratory data analysis phase to understand the extent and nature of missing data, which is crucial for subsequent modeling steps. It can identify different patterns of missingness across variables and analyze their relationship with a target variable, helping to decide on an appropriate imputation strategy.
| Parameter | Description |
|---|---|
| table | Specifies the input table for the analysis. This table should contain the data for which you want to analyze missing value patterns. |
| casOut | Specifies the output table to store the results of the missing pattern analysis. This is a required parameter. |
| inputs | Specifies the list of variables to be included in the analysis. If not specified, all numeric and character variables from the input table are used. |
| nominals | Specifies which of the input variables should be treated as nominal (categorical). This affects how statistics are calculated for these variables. |
| target | Specifies a target variable. When a target is provided, the action analyzes the relationship between missing value patterns and the target variable's distribution. |
| freq | Specifies a frequency variable. Each observation in the input table is treated as if it appears n times, where n is the value of the frequency variable for that observation. |
| distinctCountLimit | Sets a limit on the number of distinct values for frequency counting. If this limit is exceeded, the action may switch to an estimation algorithm (Misra-Gries) or abort. |
| ecdfTolerance | Specifies the tolerance for the empirical cumulative distribution function (ECDF), used by the quantile sketch algorithm for robust statistics. |
| misraGries | When set to TRUE, enables the use of the Misra-Gries algorithm for frequency estimation if the distinct count limit is surpassed. |
Creating a Sample Dataset with Missing Values
This SAS code creates a sample dataset named 'sample_data_missing' in the 'casuser' caslib. The dataset includes several variables with intentionally placed missing values (.) to demonstrate how the analyzeMissingPatterns action works.
| 1 | DATA casuser.sample_data_missing; |
| 2 | INPUT var1 var2 $ var3 var4 target; |
| 3 | CARDS; |
| 4 | 1 10 A 100 1 |
| 5 | 2 . B 200 0 |
| 6 | 3 30 . 300 1 |
| 7 | 4 40 C . 0 |
| 8 | 5 . D 500 1 |
| 9 | 6 60 . 600 0 |
| 10 | 7 70 E . 1 |
| 11 | ; |
| 12 | RUN; |
Examples
This example performs a fundamental missing pattern analysis on the 'sample_data_missing' table. The results, which include tables detailing missing counts and patterns, are saved to a CAS table named 'missing_patterns_summary'.
| 1 | PROC CAS; |
| 2 | dataSciencePilot.analyzeMissingPatterns / |
| 3 | TABLE={name='sample_data_missing'}, |
| 4 | casOut={name='missing_patterns_summary', replace=true}; |
| 5 | RUN; |
| 6 | QUIT; |
The action generates several result tables. The 'MissingPatterns' table outlines the different combinations of missing values found. The 'MissingCounts' table provides the count and percentage of missing values for each variable. 'NumVarInfo' and 'CharVarInfo' provide descriptive statistics for numeric and character variables, respectively.
This detailed example analyzes missing value patterns in relation to a specific target variable ('target'). It explicitly defines 'var3' as a nominal variable and focuses the analysis on a specified list of input variables. This approach helps in understanding if the missingness of data is correlated with the outcome variable.
| 1 | PROC CAS; |
| 2 | dataSciencePilot.analyzeMissingPatterns / |
| 3 | TABLE={name='sample_data_missing'}, |
| 4 | inputs={{name='var1'}, {name='var2'}, {name='var3'}, {name='var4'}}, |
| 5 | nominals={'var3'}, |
| 6 | target='target', |
| 7 | casOut={name='missing_patterns_details', replace=true}; |
| 8 | RUN; |
| 9 | QUIT; |
The output includes several analytical tables. Notably, the 'TargetCounts' table shows the frequency distribution of the target variable for each identified missing value pattern. The 'TargetMeans' table provides the mean of the target variable for each pattern. These results are crucial for determining if the missing data mechanism is Missing Not At Random (MNAR) with respect to the target.
FAQ
Associated Scenarios
Use Case
Standard Case: Customer Churn Prediction - Impact of Missing Demographic Data
A telecom company wants to understand why customers are churning. They suspect that missing information in their CRM (like income or age) is not random and might be correlated w...
Use Case
Performance Case: Analyzing Missing Sensor Readings in High-Volume IoT Data
A manufacturing plant uses thousands of IoT sensors to monitor equipment. Some sensors intermittently fail to report data. The goal is to analyze these missing data patterns at ...
Use Case
Edge Case: Handling Dropouts and Sparse Data in Clinical Trial Analysis
In a clinical trial, patients may drop out, leading to missing data for all subsequent visits. One specific biomarker is experimental and fails to record for most patients. This...