物联网 | TensorFlow 气象站预测降雨强度

⭐ 输入电子表格 ID 和范围以将气象站的最新发现附加到电子表格中。

https://docs.google.com/spreadsheets/d/spreadsheetId/edit#gid=0

// 输入您的电子表格 ID：
$spreadsheetId ='';
// 输入将追加新值的范围（第一行）（8 行） :
$range ='A1:H1';
// 将气象站的最新发现添加到电子表格中。
$values =[
 [$variables_from_module["wd" ], $variables_from_module["a_ws"], $variables_from_module["m_ws"], $variables_from_module["1_rf"], $variables_from_module["24_rf"], $variables_from_module["tem"], $variables_from_module["hum $variables_from_module["b_pr"]]
];
$body =new Google_Service_Sheets_ValueRange([
 'values' => $values
]);
$params =[
 'valueInputOption' => "RAW"
];

...

 $result =$service->spreadsheets_values->append( $spreadsheetId, $range, $body, $params);
 printf("

%d 个单元格追加。", $result->getUpdates()->getUpdatedCells()); 

💻 account_verification_token.php

⭐ 通过创建的 AuthUrl 授权 Web 应用程序后 , 输入给定的验证码以设置访问令牌。

$account_verification_token ="<输入令牌>"; // 授权后输入验证码。 

步骤 3.1：在 Raspberry Pi（或任何服务器）上设置 Web 应用程序

创建 PHP Web 应用程序后，我决定在我的 Raspberry Pi 上运行它，但您可以在任何服务器上运行该应用程序，只要它是 PHP 服务器即可。

如果您想使用树莓派，但不知道如何在树莓派上设置 LAMP 网络服务器，您可以查看本教程。

⭐首先，由于apache服务器是受保护的位置，因此使用终端将应用程序文件夹（remote_weather_station）移动到apache服务器（/var/www/html）。

sudo mv /home/pi/Downloads/remote_weather_station /var/www/html/

⭐ 由于缺少验证码，Web 应用程序在第一次打开时抛出错误。获取验证码，进入应用生成的授权链接。

⭐然后，点击转到远程气象站（不安全） .

⭐ 授予应用所需的权限。

⭐现在，复制验证码并粘贴到account_verification_token.php .

⭐ 返回 Web 应用程序。它应该显示确认消息：成功！刷新页面 .

⭐ 刷新页面后，Web 应用程序使用验证码获取访问令牌并将访问令牌保存在名为 token.json 的文件中 .因此，它不会再次抛出错误。

⭐ 要测试 PHP Web 应用程序是否准确地将新数据附加到给定的电子表格：

http://localhost/remote_weather_station/?wd=12&a_ws=8&m_ws=11&1_rf=14&24_rf=84&tem=24&hum=32&b_pr=57

第 4 步：使用 ESP8266（WiFi）读取和发送天气数据

为了在阳台上收集天气数据，我使用了 NodeMCU ESP-12E (ESP8266) 开发板和气象站。

我将其编程为每五分钟向 PHP Web 应用程序发送天气数据。

⭐ 包含所需的库并定义 WiFi 设置。

#include 
#include  
#include 
#include 
 #include 

// 定义您的 WiFi 设置。
const char *ssid ="";
const char *password =" "; 

⭐ 定义气象站设置和串行连接引脚 - D6、D5。

//定义气象站设置：
char databuffer[35];
double temp;
int transfer =0;

//定义串行连接引脚 - RX 和 TX。
SoftwareSerial Serial_1(D6, D5); // (Rx, Tx) 

⭐ 在 getBuffer() 中 功能，从气象站获取数据。

void getBuffer(){
 int index;
 for (index =0;index <35;index ++){
 if(Serial_1.available()){ 
 databuffer[index] =Serial_1.read();
 if (databuffer[0] !='c'){
 index =-1;
 }
 }
 else{
 index --;
 }
 }
} 

⭐ 调试天气数据并创建链接。

String weather_data ="wd=" + String(WindDirection()) + "&a_ws=" + String(WindSpeedAverage()) + "&m_ws=" + String(WindSpeedMax()) + "&1_rf=" + String(RainfallOneHour()) + "&24_rf=" + String(RainfallOneDay()) + "&tem=" + String(Temperature()) + "&hum=" + String(Humidity()) + "&b_pr=" + String( BarPressure());
String server ="http://192.168.1.24/remote_weather_station/?";

...

int WindDirection(){返回 transCharToInt(databuffer,1,3); } // 风向（度）

float WindSpeedAverage(){ temp =0.44704 * transCharToInt(databuffer,5,7);返回温度； } // 平均空速（1 分钟）

float WindSpeedMax(){ temp =0.44704 * transCharToInt(databuffer,9,11);返回温度； } //最大空气速度（5分钟）

float Temperature(){ temp =(transCharToInt(databuffer,13,15) - 32.00) * 5.00 / 9.00;返回温度； } // 温度 ("C")

float RainfallOneHour(){ temp =transCharToInt(databuffer,17,19) * 25.40 * 0.01;返回温度； } // 降雨量（1 小时）

float RainfallOneDay(){ temp =transCharToInt(databuffer,21,23) * 25.40 * 0.01;返回温度； } // 降雨量（24 小时）

int Humidity(){ return transCharToInt(databuffer,25,26); } // 湿度 (%)

float BarPressure(){ temp =transCharToInt(databuffer,28,32);返回温度 / 10.00； } // Barometric Pressure (hPA) 

⭐ Send data packets every 5 minutes to Raspberry Pi (or any server).

transferring++; Serial.println("Time => " + String(transferring) + "s / " + String(int(5*60)) + "s\n\n");
 if(transferring ==5*60){
 // Create the HTTP object to make a request to the server. 
 HTTPClient http; 
 http.begin(server + weather_data);
 int httpCode =http.GET(); 
 String payload =http.getString(); 
 Serial.println("Data Send...\nHTTP Code => " + String(httpCode) + "\nServer Response => " + payload + "\n\n"); 
 http.end();
 transferring =0; 
 }
 // Wait 1 second...
 delay(1000); 

⭐ After uploading the code to the NodeMCU ESP-12E (ESP8266) development board, it displays weather data every second on the serial monitor and sends data packets every five minutes (300 seconds) to the PHP web application.

Connection is successful!

...

Weather Data => wd=0&a_ws=0.00&m_ws=0.00&1_rf=0.00&24_rf=0.00&tem=21.67&hum=29&b_pr=1016.70
Buffer => c000s000g000t071r000p000h29b10167*3
Time => 299s / 300s


Weather Data => wd=0&a_ws=0.00&m_ws=0.00&1_rf=0.00&24_rf=0.00&tem=21.67&hum=29&b_pr=1016.70
Buffer => c000s000g000t071r000p000h29b10167*3
Time => 300s / 300s

... 

⭐ Then, it shows the response from the server.

...

Data Send...
HTTP Code => 200
Server Response => Token Found!

8 cells appended.

... 

Step 4.1:Reading and sending weather data with SIM808 (GPRS)

To collect weather data in my backyard, I used a SIM808 shield for Arduino Uno if the distance between the weather station and my router is too far away.

Most of the code is the same and covered in the previous step aside from the parts below.

⭐ Include required libraries and define the sim808 object.

For SIM808 GPS/GPRS/GSM Shield | Download

#include 
#include 

// Define the sim808.
DFRobot_SIM808 sim808(&Serial); 

⭐ Initialize the SIM808 module and continue if it is working accurately.

//******** Initialize sim808 module *************
 while(!sim808.init()) {
 delay(1000);
 Serial.print("Sim808 init error\r\n");
 }
 delay(2000);
 // Continue if the SIM808 Module is working accurately.
 Serial.println("Sim808 init success");
 delay(5000); 

⭐ Send data packets every 5 minutes to the server by establishing a TCP connection to make a GET request.

transferring++; Serial.println("Time => " + String(transferring) + "s / " + String(int(5*60)) + "s\n\n");
 if(transferring ==5*60){
 //*********** Attempt DHCP *******************
 while(!sim808.join(F("cmnet"))) {
 Serial.println("Sim808 join network error!");
 delay(2000);
 }
 //************ Successful DHCP ****************
 delay(5000);
 Serial.println("Successful DHCP");
 //*********** Establish a TCP connection ************
 if(!sim808.connect(TCP,"192.168.1.24", 80)) { // Change it with your server.
 Serial.println("Connection Error");
 }else{
 Serial.println("Connection OK");
 }
 delay(2000);
 
 ... 

⭐ After creating the line string, convert it from string to char array to make an HTTP Get Request with the SIM808.

...
 
 String line ="GET /remote_weather_station/?" + weather_data_1 + weather_data_2 + weather_data_3 + " HTTP/1.0\r\n\r\n";
 Serial.println(line);
 char buffer[512];
 // Convert the line from string to char array to make an HTTP Get Request with the SIM808.
 char web_hook[110];
 String_to_Char(line, 110, web_hook);
 sim808.send(web_hook, sizeof(web_hook)-1);
 while (true) {
 int ret =sim808.recv(buffer, sizeof(buffer)-1);
 if (ret <=0){
 Serial.println("Fetch Over...");
 break; 
 }
 
 ... 

⭐ After uploading the code to the SIM808 shield, it displays weather data every second on the serial monitor and sends data packets every five minutes (300 seconds) to the PHP web application.

Step 4.2:Creating the weather data set for more than three months

After finishing coding, I started to collate weather data on Google Sheets every five minutes for more than three months to build a neural network model and make predictions on the rainfall intensity.

Collected Weather Data:

• Wind Direction (deg)
• Average Wind Speed (m/s)
• Max Wind Speed (m/s)
• One-Hour Rainfall (mm)
• 24-Hours Rainfall (mm)
• Temperature (°C)
• Humidity (%)
• Barometric Pressure (hPa)

Then, I downloaded the spreadsheet as Remote Weather Station.csv , consisting of 32219 rows as my preliminary local weather data set. I am still collating weather data to improve my data set and model :)

Step 5:Building an Artificial Neural Network (ANN) with TensorFlow

When I completed collating my preliminary local weather data set, I started to work on my artificial neural network (ANN) model to make predictions on the rainfall intensity.

I decided to create my neural network model with TensorFlow in Python. So, first of all, I followed the steps below to grasp a better understanding of the weather data:

• Data Visualization
• Data Scaling (Normalizing)
• Data Preprocessing
• Data Splitting

After applying these steps, I decided to use my neural network model to classify different rainfall intensity classes theoretically assigned as labels (outputs) for each input (row). I created my classes according to the rate of precipitation, which depends on the considered time.

The following categories are used to classify rainfall intensity by the rate of precipitation (rainfall):

• Light rain — when the precipitation rate is <2.5 mm per hour
• Moderate rain — when the precipitation rate is between 2.5 mm and 7.6 mm per hour
• Heavy rain — when the precipitation rate is between 7.6 mm and 50 mm per hour
• Violent rain — when the precipitation rate is> 50 mm per hour

According to the precipitation (rainfall) rates, I preprocessed the weather data to assign one of these five classes for each input as its label:

• 0 (None)
• 1 (Light Rain)
• 2 (Moderate Rain)
• 3 (Heavy Rain)
• 4 (Violent Rain)

After scaling (normalizing) and preprocessing the weather data, I elicited seven input variables and one label for each reading, classified with the five mentioned classes. Then, I built an artificial neural network model with TensorFlow to obtain the best possible results and predictions with my preliminary data set.

Layers:

• 7 [Input]
• 16 [Hidden]
• 32 [Hidden]
• 64 [Hidden]
• 128 [Hidden]
• 5 [Output]

To execute all steps above, I created a class named Weather_Station in Python after including the required libraries:

import tensorflow as tf
from tensorflow import keras
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd 

Subsequently, I will discuss coding in Python for each step I mentioned above.

Also, you can download IoT_weather_station_neural_network.py to inspect coding.

Step 5.1:Visualizing and scaling (normalizing) the weather data

Before diving in to build a model, it is important to understand the weather data to pass the model appropriately formatted data.

So, in this step, I will show you how to visualize weather data and scale (normalize) in Python.

⭐ First of all, read the weather data from Remote Weather Station.csv .

csv_path ="E:\PYTHON\Remote Weather Station.csv"
df =pd.read_csv(csv_path) 

⭐ In the graphics() function, visualize the requested columns from the weather data set by using the Matplotlib library.

def graphics(self, column_1, column_2, xlabel, ylabel):
 # Show requested columns from the data set:
 plt.style.use("dark_background")
 plt.gcf().canvas.set_window_title('IoT Weather Station')
 plt.hist2d(self.df[column_1], self.df[column_2])
 plt.colorbar()
 plt.xlabel(xlabel)
 plt.ylabel(ylabel)
 plt.title(xlabel)
 plt.show() 

⭐ In the data_visualization() function, inspect all columns before scaling weather data to build a model with appropriately formatted data.

def data_visualization(self):
 # Inspect requested columns to build a model with appropriately formatted data:
 self.graphics('WD', '1h_RF', 'Wind Direction (deg)', 'One-Hour Rainfall (mm)')
 self.graphics('Av_WS', '1h_RF', 'Average Wind Speed (m/s)', 'One-Hour Rainfall (mm)')
 self.graphics('Mx_WS', '1h_RF', 'Maximum Wind Speed (m/s)', 'One-Hour Rainfall (mm)')
 self.graphics('24h_RF', '1h_RF', '24-Hour Rainfall (mm)', 'One-Hour Rainfall (mm)')
 self.graphics('Tem', '1h_RF', 'Temperature (°C)', 'One-Hour Rainfall (mm)')
 self.graphics('Hum', '1h_RF', 'Humidity (%)', 'One-Hour Rainfall (mm)')
 self.graphics('b_PR', '1h_RF', 'Barometric Pressure (hPA)', 'One-Hour Rainfall (mm)')
 

After visualizing weather data, I scaled (normalized) each column to format it properly.

Normally, each row (reading) looked like this before scaling:

• 315, 0.45, 0, 0.51, 0.51, 22.78, 65, 1005.5

⭐ In the scale_data() function:

⭐ The wind direction in units of degrees and angles do not make good model inputs. 360° and 0° should be close to each other and wrap around smoothly. The direction should not matter if the wind is not blowing. Thus, convert the wind direction and velocity columns to a wind vector to interpret them easily with a neural network model.

def scale_data(self):
 # Wind Direction and Speed:
 wv =self.df.pop('Av_WS')
 max_wv =self.df.pop('Mx_WS')
 # Convert to radians.
 wd_rad =self.df.pop('WD')*np.pi / 180
 # Calculate the wind x and y components.
 self.df['scaled_WX'] =wv*np.cos(wd_rad)
 self.df['scaled_WY'] =wv*np.sin(wd_rad)
 # Calculate the max wind x and y components.
 self.df['scaled_max_WX'] =max_wv*np.cos(wd_rad)
 self.df['scaled_max_WY'] =max_wv*np.sin(wd_rad)
 
 ... 

⭐ For the remaining columns except for 1h_RF , divide them into average reading values to scale (normalize) and format.

...

 # Temperature:
 tem =self.df.pop('Tem')
 self.df['scaled_Tem'] =tem / 25
 # Humidity:
 hum =self.df.pop('Hum')
 self.df['scaled_Hum'] =hum / 70 
 # Barometric Pressure:
 bPR =self.df.pop('b_PR')
 self.df["scaled_bPR"] =bPR / 1013 
 # 24 Hour Rainfall (Approx.)
 rain_24 =self.df.pop('24h_RF')
 self.df['scaled_24h_RF'] =rain_24 / 24 

After completing scaling (normalizing), I extracted these new columns from the weather data set:

• scaled_WX
• scaled_WY
• scaled_max_WX
• scaled_max_WY
• scaled_Tem
• scaled_Hum
• scaled_bPR
• scaled_24h_RF

Step 5.2:Training the model (ANN) on the rainfall intensity classes

Before building and training a neural network model in TensorFlow, I needed to create the input array and the label array by preprocessing the scaled and normalized weather data set.

⭐ In the create_input_and_label() function:

⭐ Firstly, append each input element as a NumPy array to the input array and convert it to a NumPy array by using the asarray() function.

⭐ Each input element includes seven variables [shape=(7, )]:

• [scaled_WX, scaled_WY, scaled_max_WX, scaled_max_WY, scaled_Tem, scaled_Hum, scaled_bPR]

def create_input_and_label(self):
 n =len(self.df)
 # Create the input array using the scaled variables:
 for i in range(n):
 self.input.append(np.array([self.df['scaled_WX'][i], self.df['scaled_WY'][i], self.df['scaled_max_WX'][i], self.df['scaled_max_WY'][i], self.df['scaled_Tem'][i], self.df['scaled_Hum'][i], self.df['scaled_bPR'][i]]))
 self.input =np.asarray(self.input)
 
 ... 

⭐ Then, evaluate the approximate rainfall (precipitation) rate for each reading (row).

approx_RF_rate =(1h_RF + scaled_24h_RF) * 100

⭐ According to the rainfall rate, assign a class [0 - 4] for each input element and append them to the label array.

...
 
 for i in range(n):
 _class =0
 # Evaluate the approximate rainfall rate:
 approx_RF_rate =(self.df['1h_RF'][i] + self.df['scaled_24h_RF'][i]) * 100
 # As labels, assign classes of rainfall intensity according to the approximate rainfall rate (mm):
 if approx_RF_rate ==0:
 _class =0
 elif approx_RF_rate <2.5:
 _class =1
 elif 2.5  _class =2
 elif 7.6  _class =3
 else:
 _class =4
 self.label.append(_class)
 self.label =np.asarray(self.label) 

After preprocessing the scaled weather data to create input and label arrays, I split them as training (60%) and test (40%) data sets:

def split_data(self):
 n =len(self.df)
 # (60%, 40%) - (training, test)
 self.train_input =self.input[0:int(n*0.6)]
 self.test_input =self.input[int(n*0.6):]
 self.train_label =self.label[0:int(n*0.6)]
 self.test_label =self.label[int(n*0.6):] 

Then, I built my artificial neural network (ANN) model by using Keras and trained it with the training data set for nineteen epochs.

You can inspect these tutorials to learn about activation functions, loss functions, epochs, etc.

def build_and_train_model(self):
 # Build the neural network:
 self.model =keras.Sequential([
 keras.Input(shape=(7,)),
 keras.layers.Dense(16, activation='relu'),
 keras.layers.Dense(32, activation='relu'),
 keras.layers.Dense(64, activation='relu'),
 keras.layers.Dense(128, activation='relu'),
 keras.layers.Dense(5, activation='softmax')
 ])
 # Compile:
 self.model.compile(optimizer='adam', loss="sparse_categorical_crossentropy", metrics=['accuracy'])
 # Train:
 self.model.fit(self.train_input, self.train_label, epochs=19)
 
 ... 

After training with the preliminary training data set, the accuracy of the model is between 0.83 and 0.85 .

Step 5.3:Evaluating the model and making predictions on the rainfall intensity

After building and training my artificial neural network model, I tested its accuracy by using the preliminary testing data set.

For now, the evaluated accuracy of the model is between 0.72 and 0.73 due to inadequate testing data set overfitting the model. However, I am still collecting weather data to improve the model accuracy.

...
 
 # Test the accuracy:
 print("\n\nModel Evaluation:")
 test_loss, test_acc =self.model.evaluate(self.test_input, self.test_label) 
 print("Evaluated Accuracy:", test_acc) 

Then, I used my neural network model to make predictions on the rainfall intensity with a given prediction array consisting of readings from the weather station after the training of the model. As a starting point, the model works fine :)

The model predicts possibilities of labels for each input element as an array of 5 numbers. They represent the model's "confidence" that the given input element corresponds to each of the five different classes of rainfall intensity [0 - 4].

⭐ In the make_prediction() function, make a prediction for each input element in a given array and get the most accurate label [0 - 4] by using the argmax() function to display its class name.

def make_prediction(self, pre_array):
 print("\n\nModel Predictions:\n")
 prediction =self.model.predict(pre_array)
 for i in range(len(pre_array)):
 print("Prediction => ", self.class_names[np.argmax(prediction[i])]) 

Prediction Inputs:

• [0, 0, 0.31819805, 0.31819805, 0.6988, 0.81498571, 0.99349753]
• [0, -0, 0, -0, 0.8444, 1, 0.96835143]
• [0, 0, 0.45, 0, 0.87577, 0.95857143, 1.00128332]
• [-0, -0, -0, -0, 0.8224, 1.05714286, 0.99279368]

Prediction Outputs:

• 0 [None]
• 3 [Heavy Rain]
• 4 [Violent Rain]
• 4 [Violent Rain]

Connections and Adjustments (ESP8266)

// Connections
// NodeMCU ESP-12E (ESP8266) :
// Weather Station
// VV --------------------------- 5V
// D5 --------------------------- RX
// D6 --------------------------- TX
// G --------------------------- GND 

I connected the sensor (converter) board to the NodeMCU ESP-12E (ESP8266) development board and fastened them to a plastic box while collecting weather data at my balcony.

Connections and Adjustments (SIM808)

// Connections
// Arduino Uno:
// SIM808 GPS/GPRS/GSM Shield For Arduino
// D0 --------------------------- RX
// D1 --------------------------- TX
// D12 --------------------------- POWER 
// Weather Station
// 5V --------------------------- 5V
// D5 --------------------------- RX
// D6 --------------------------- TX
// GND --------------------------- GND 

⭐ Note:D0, D1, D12 pins are occupied by the SIM808 GPS/GPRS/GSM Shield.

⭐ Connect an external battery (7-23V) for the SIM808 module to work properly.

⭐ Attach the GPS antenna and the GSM antenna to the SIM808 shield.

⭐ Insert a SIM card into the SIM slot on the SIM808 shield.

⭐ Before uploading the code, set the function switch on the shield to None (1).

⭐ Upload the code.

⭐ Then, set the function switch to Arduino (3).

⭐ Press the Boot button on the shield until seeing the Net indicator LED flashing every 1 second and wait for the SIM card to register the network - the Net indicator LED will slowly flash every 3 seconds.

⭐ Click here to get more information about the SIM808 GSM/GPS/GPRS Shield.

After setting up the SIM808 shield with Arduino Uno, I connected the sensor (converter) board to it.

I placed all components into a plastic box to collect weather data outside, even in extreme weather conditions, since the sensor board is not weather-proof.

Then, I fastened the plastic box to the weather station.

Videos and Conclusion

After completing coding and assembling the weather station, I collated local weather data for more than three months every five minutes to build my neural network model successfully.

I am still collecting weather data at my balcony and backyard to improve my neural network model and its accuracy :)

Further Discussions

☔ 💧 Since we need local weather data sets to get more accurate predictions on weather phenomena, budget-friendly NN-enabled weather stations like this can be placed on farms and greenhouses to avert the detrimental effects of excessive rainfall on agriculture.

☔ 💧 As early warning systems working with neural networks, we can use them simultaneously to create a swarm of weather stations communicating and feeding each other to forecast rainfall intensity precisely in local areas.

☔ 💧 Also, we can utilize that information for assessing:

• vital water resources,
• agriculture,
• crop productivity,
• ecosystems,
• hydrology.

References

[1] Extreme weather - heavy rainfall , NIWA, https://niwa.co.nz/natural-hazards/extreme-weather-heavy-rainfall

[2] University of Illinois at Urbana-Champaign, News Bureau. "Excessive rainfall as damaging to corn yield as extreme heat, drought. " ScienceDaily. ScienceDaily, 30 April 2019. www.sciencedaily.com/releases/2019/04/190430121744.htm.

[3] Andrew Culclasure, Using Neural Networks to Provide Local Weather Forecasts , Georgia Southern University, Spring 2013, 29https://digitalcommons.georgiasouthern.edu/cgi/viewcontent.cgi?article=1031&context=etd.