Business Context
An industrial plant generates massive CSV logs from IoT sensors stored in a data lake. The Data Engineer needs to inspect the schema of a new daily log file ('sensor_log_big.csv') to map the columns correctly before attempting to load it into memory, as loading the full file is resource-intensive and time-consuming.
Data Preparation
Simulation of a large sensor dataset exported to a CSV file in the CASUSER library.
Copied!
| 1 | |
| 2 | DATA casuser.sensor_temp; |
| 3 | DO i=1 to 1000; |
| 4 | SensorID=i; |
| 5 | Reading_Value=rand('normal', 50, 5); |
| 6 | Reading_Time=datetime(); |
| 7 | OUTPUT; |
| 8 | END; |
| 9 | |
| 10 | RUN; |
| 11 | |
| 12 | PROC CAS; |
| 13 | TABLE.save TABLE={name='sensor_temp', caslib='casuser'} name='sensor_log_big.csv' replace=true; |
| 14 | TABLE.dropTable name='sensor_temp' caslib='casuser'; |
| 15 | |
| 16 | QUIT; |
| 17 |
Étapes de réalisation
1
Inspect the file directly on disk (CASLIB source) without loading it into RAM.
Copied!
| 1 | |
| 2 | PROC CAS; |
| 3 | TABLE.columnInfo RESULT=r / TABLE={name='sensor_log_big.csv', caslib='casuser'}; |
| 4 | PRINT r; |
| 5 | |
| 6 | QUIT; |
| 7 |
Expected Result
The action successfully reads the header of the 'sensor_log_big.csv' file from the storage (Caslib) and displays the column names and inferred types (e.g., Reading_Value as Double) without creating an in-memory table.