in Uncategorized

 I recently completed a data-logging project that was long overdue. My dad is building a double-walled, super-insulated house, and wanted me to track temperature/humidity in multiple places throughout the structure. In doing so, we plan to calculate overall efficiency, as well as track down areas to recover wasted heat (such as a 120-degree attic during the summer, or the basement near the wood-stove in the winter). We plan to capture this neglected heat via a combination of heat pumps and a closed-loop heat recovery system in the attic, but that is a story for another day.

A little over a year ago, I wrote a simple Python script that integrated with an Arduino to collect temp and humidity figures. It used the GData API to write to a spreadsheet, where the figures were then easily analyzed. In short, it sucked. I learned a lot through the process however, including the horrors of TKinter. I also learned that it is possible to write very, very bad Python code. There were many bugs in the implementation, and I decided to swap to another platform for data collection instead of chasing down timing issues in the serial bus and DHT sensors. I then wrote a Labview program coupled with Labview Interface for Arduino to collect the data, display it on a front panel, and append it to a text file (Labview’s first problem, of many). This system worked, but for the sake of extensibility and sheer stubbornness, I moved it back to a Python-based version. Here’s how to do it.

Step 1: Arduino Sketch 

I simply used Adafruit’s DHT library, and wrote a sketch to read from seven sensors. Since the system is implemented in a house, each sensor is connected to the Arduino with 40-100′ of wire. At first I had an issue with latency while reading sensor data over such a long run, but solved it after hours of troubleshooting. 

HUGE HINT: For long runs, skip 1 pin on the Arduino when iterating through sensors. Ex: Sensor 1 is pin 1, Sensor 2 is pin 3, Sensor 3 is pin 5, etc. 

Every 5 minutes the sketch pings each sensor and writes the values on one line to the serial port. Temperature and humidity per sensor are separated with colons, while sensors are separated with semicolons.

Ex: temp1:humidity1;temp2:humidity2; tempN, humidityN

To learn how to install libraries, check out the Arduino documentation here.

//Sketch to print temp/humidity
// Include the DHT library
#include "DHT.h"

// Define PIN vars
#define DHTPIN 2
#define DHTPIN2 4
#define DHTPIN3 6
#define DHTPIN4 8
#define DHTPIN5 10
#define DHTPIN6 12
#define DHTPIN7 3
#define DHTTYPE DHT22

// Initialize dht and dht2 vars
DHT dht1(DHTPIN, DHTTYPE);
DHT dht2(DHTPIN2, DHTTYPE);
DHT dht3(DHTPIN3, DHTTYPE);
DHT dht4(DHTPIN4, DHTTYPE);
DHT dht5(DHTPIN5, DHTTYPE);
DHT dht6(DHTPIN6, DHTTYPE);
DHT dht7(DHTPIN7, DHTTYPE);

void setup() {
  Serial.begin(9600); 
  dht1.begin();
  dht2.begin();
  dht3.begin();
  dht4.begin();
  dht5.begin();
  dht6.begin();
  dht7.begin();
}

void loop() {
  float h1 = dht1.readHumidity();
  float t1 = dht1.readTemperature(); 
  float h2 = dht2.readHumidity();
  float t2 = dht2.readTemperature(); 
  float h3 = dht3.readHumidity();
  float t3 = dht3.readTemperature(); 
  float h4 = dht4.readHumidity();
  float t4 = dht4.readTemperature(); 
  float h5 = dht5.readHumidity();
  float t5 = dht5.readTemperature(); 
  float h6 = dht6.readHumidity();
  float t6 = dht6.readTemperature(); 
  float h7 = dht7.readHumidity();
  float t7 = dht7.readTemperature(); 

   // Check if any reads failed and exit early (to try again).
  if (isnan(h1) || isnan(t1) || isnan(h2) || isnan(t2) || isnan(t3) || isnan(h3) 
       || isnan(t4) || isnan(h4) || isnan(t5) || isnan(h5) || isnan(t6) || isnan(h6)
       || isnan(t7) || isnan(h7) {
    Serial.println("Sensor read fail. Retrying....");
  }
  else {
    // Separate sensors with a semicolon, separate temp/humidity with a colon
    Serial.print(cToF(t1));
    Serial.print(":");  
    Serial.print(h1);
    Serial.print(";");
    Serial.print(cToF(t2));
    Serial.print(":");  
    Serial.print(h2);
    Serial.print(";");
    Serial.print(cToF(t3));
    Serial.print(":");  
    Serial.print(h3);
    Serial.print(";");
    Serial.print(cToF(t4));
    Serial.print(":");  
    Serial.print(h4);
    Serial.print(";");
    Serial.print(cToF(t5));
    Serial.print(":");  
    Serial.print(h5);
    Serial.print(";");
    Serial.print(cToF(t6));
    Serial.print(":");  
    Serial.print(h6);
    Serial.print(";");
    Serial.print(cToF(t7));
    Serial.print(":");  
    Serial.print(h7);
    Serial.println();
    delay(60000);
  }
}

float cToF(float x){
  float y;
  y = (x*1.8)+32;
  return y;
}

 

Step 2: Python Middleware

The next step in the process includes setting up a Python script to act as middleware. Its job is simply to grab data in the serial receive buffer, grab current date/time, acquire local weather data from Weather Underground’s API, parse all fore-mentioned data into a list, and write the list to a daily CSV file/ an Xively account.

A brief walk-through of the script includes getting the Weather Underground Data:

def getWunderground(geolookup):
  """ Geolookup = weather underground geolookup url: (http://www.wunderground.com/weather/api/d/docs?d=data/geolookup),
  determines if program has hit the API within the last 30 minutes. If yes, returns values of last API call.
  If no, it tries to call the API and returns a list of the current weather, current temp, rel humidity, and UV index.
  If there is no connection to the internet/ wunderground API, function returns a list with all values populated with "no connection".
  """
  dataFromWunderground = []
  global wuLastUpdate
  global wuData

  timeDelta = datetime.datetime.now()-wuLastUpdate
  print timeDelta
  if timeDelta >= datetime.timedelta(minutes=30):
    try:
      f = urllib2.urlopen(geolookup)
      json_string = f.read()
      parsed_json = json.loads(json_string)
      weather = parsed_json["current_observation"]["weather"]
      temp_f = parsed_json["current_observation"]["temp_f"]
      relative_humidity = parsed_json["current_observation"]["relative_humidity"]
      uv = parsed_json["current_observation"]["UV"]
      #pressure_in = parsed_json["current_observation"]["pressure_in"]
      #wind_degrees = parsed_json["current_observation"]["wind_degrees"]
      #wind_mph = parsed_json["current_observation"]["wind_mph"]
      f.close()
      dataFromWunderground = [str(weather), str(uv), str(temp_f), str(relative_humidity[:-1]) ]
      wuData = dataFromWunderground
      wuLastUpdate = datetime.datetime.now()
    except:
      for i in range(0,4):
        dataFromWunderground.append("No connection")
      wuData = dataFromWunderground
      wuLastUpdate = datetime.datetime.now()
  else:
    dataFromWunderground = wuData
  return dataFromWunderground

 

 

Next, script gets current date and time

def getDateTime():
  """Function grabs current time and date, then returns values in a 2-element list. """
  timeNow = time.strftime("%H:%M:%S")
  dateToday = time.strftime("%m/%d/%y")
  return [dateToday, timeNow]

 

Once main loop grabs the array of data, it appends everything to a local CSV file on the host computer (or Raspberry Pi, since they run Python out of the box). If no CSV file is found, this function creates one and appends to it.

def writeToCsv(datalist):
  """ function writes datalist values to a csv file. If daily csv file exists already, 
  list values are simply appended to end of file. If it does not, function creates the file, 
  then appends values. 
  """ 
  global csv_success

  header = ["date", "time", "weather", "uv", "exterior temp", "exterior humidity", "temp1", "hum2", "temp2", "hum2", "temp3", "hum3", "temp4", "hum4", "temp5", "hum5", "temp6", "hum6", "temp7", "hum7", "iterations","\n"]

  fileName = str(time.strftime("%m_%d_%y_")+ "log.csv")
  if os.path.exists(fileName):
    f = open(fileName, "a")
  else:
    f = open(fileName, "a+")
    for element in header:
      f.write(element + ",")
    f.write("\n")

  for element in datalist:
    if type(element)==str:
      f.write(element + ",")
    if type(element) == list:
      for i in element:
        f.write(i + ",")
  f.write("\n")
  f.close()
  csv_success = True

 

 

One of the last steps involves writing the data to various user-defined streams inside an Xively feed. 

def printXively(dataList):
  """datalist = list of weather underground data, current date and time, and sensor values. Function 
    writes each element to the proper Xively data stream."""
  global xively_success

  XIVELY_API_KEY = YOUR_API_KEY_HERE
  XIVELY_FEED_ID = YOUR_XIVELY_FEED_ID_HERE
  api = xively.XivelyAPIClient(XIVELY_API_KEY)
  feed = api.feeds.get(XIVELY_FEED_ID)
  now = datetime.datetime.now()
  feed.datastreams = [
    xively.Datastream(id="uv", current_value=dataList[3], at=now),
    xively.Datastream(id="weather", current_value=dataList[2], at=now),
    xively.Datastream(id="exterior_temp", current_value=dataList[4], at=now),
    xively.Datastream(id="exterior_humidity", current_value=dataList[5], at=now),
    xively.Datastream(id="temp1", current_value=dataList[6][0], at=now),
    xively.Datastream(id="humidity1", current_value=dataList[6][1], at=now),
    xively.Datastream(id="temp2", current_value=dataList[7][0], at=now),
    xively.Datastream(id="humidity2", current_value=dataList[7][1], at=now),
    xively.Datastream(id="temp3", current_value=dataList[8][0], at=now),
    xively.Datastream(id="humidity3", current_value=dataList[8][1], at=now),
    xively.Datastream(id="temp4", current_value=dataList[9][0], at=now),
    xively.Datastream(id="humidity4", current_value=dataList[9][1], at=now),
    xively.Datastream(id="temp5", current_value=dataList[10][0], at=now),
    xively.Datastream(id="humidity5", current_value=dataList[10][1], at=now),
    xively.Datastream(id="temp6", current_value=dataList[11][0], at=now),
    xively.Datastream(id="humidity6", current_value=dataList[11][1], at=now),
    xively.Datastream(id="temp7", current_value=dataList[12][0], at=now),
    xively.Datastream(id="humidity7", current_value=dataList[12][1], at=now),

  ]

  feed.update()
  xively_success = True

 

 

Last of all, the main loop. As mentioned before, it scans the serial receive buffer and executes. Once finished, the program sleeps for 4.5 minutes to conserve system resources. The Arduino produces data every 5 minutes, and the computer running the middleware sleeps 90% of that time. Timing is naturally set upon first execution.

def mainLoop():
  """mainLoop checks the serial receive buffer for activity. If it sees anything, it grabs 
  current date/time and weather underground data. It grabs the semi-colon/colon separated values in the 
  serial buffer, appends them to the array already containing date/time/wunderground data, and tries to 
  write array to a CSV and Xively. It then increases the iteration count by 1, and puts the program to sleep
  for 4.5 minutes (to save resources on host computer). The arduino sends data every 5 minutes, so a 4.5 minute
  sleep cycle is plenty long enough to conserve resouces without mistiming data transfer. """ 

  global xively_success
  global csv_success
  global iterations
  global geolookup

  data = []
  ser = serial.Serial("/dev/tty.usbmodem1421", 9600)
  print "Serial Initialized"

  while 1:
    if ser.inWaiting():
      datetimeData = getDateTime()
      weatherData = getWunderground()

      for i in datetimeData:
        data.append(i)
      for i in weatherData:
        data.append(i)
      val = ser.readline().strip('\n\r').split(';')

      for i in range(0, len(val)):
        sensorData = val[i].split(':')
        data.append(sensorData)

      data.append(str(iterations))

      try: 
        writeToCsv(data)
        csv_success = True
      except:
        csv_success = False
      try:
        printXively(data)
        xively_success = True
      except:
        try:
          printXively(data)
          xively_success = True
        except:
          xively_success = False

      print status(xively_success, csv_success), ": ", data 
      iterations += 1
      data = []

      time.sleep(270)

 

 

All code can be cloned from Github here.

 

Lastly, the fun stuff you can do with Xively! Formerly Pachube/ Cosm, Xively is a great resource for DIYers, even though it is more focused on highly scalable applications. I’m using it because I don’t want to write a bunch of JS graphs for analysis, yet want to see a visual representation of what the house is doing. As seen below, highly sealed, super efficient structures are awesome.

Xively with Python, DHT22

Xively with Python, DHT22, Arduino

 

 

 

I also wrote a quick html/css/js dashboard for personal use, with real-time data from stream’s json. An awesome tutorial for doing so can be found here.

Xively Arduino DHT22 Python Raspberry Pi Dashboard

 

 

Hope this post is read by many and appreciated by some. Enjoy.

  • Harsha

    Hello Jacob, I am a Master student in germany, and i am at my final semester doing my master thesis, I have around 8 Gas sensors, and i have an Arduino mega board and a raspberry pi, I want to build a logger out of this for my thesis. is this possible? I also have a bluetooth module and a wifi dongle for the R-pi, can i transmit the info over those to my phone to create a graph or something ?

    • With those materials, it should be very straight forward to build a gas logger. I would suggest sending the data to a cloud-based service, such as Plot.ly, or spinning up your own api and web application to store, format, and display the data.

    • mpoint

      Hello Harsha.

      Jacob may have used http://www.adafruit.com/product/191 not sure? But connecting all different modules is not only lot of unecessary work but it can get buggy and open pandoras box, making the thesis project experience less pleasant and even frustrating at times. I recommend the use of light bean https://punchthrough.com/bean/ it comes with Bluetooth 4.0 low energy that is super helpful when it comes to continous data loggers. Because logging takes lots of CPU cycles This on has AT328p processor with build in temperature sensors. All you do is add your own gas sensor and send data to your Android, iOS or Windows Application using bluetooth and same blue tooth can be used to program the module. I have not used it, just yet but hope to in the near future.

  • Hunter Allen

    “Hope this post is read by many and appreciated by some.
    Enjoy.”

    Appreciated doesn’t even begin to describe how I feel about your contributions both to my own knowledge and the completion of my project. Awesome tutorials and perfect explanations…especially for those of us drowning in Arduino C afraid to dive straight into Python. Great job!!! If you posted daily, I’d definitely make sure to check out every one! Thanks again.

    P.S. I think Python is totally great (and easy) now 😀

  • Jesse Boxell

    hey mate, ive referred back to this post too many times now, im wondering how i could go about doing this into a plotly service as xively has gone corporate now.

    • Hey Jesse- I’m glad it helps. If you want to go the Plot.ly route (which is totally doable), check out their Python documentation. Instead of the printXively(data) function, I’d create a printPlotly(data) function which formats and sends the data. Plot.ly also has a streaming service, which you could use to build a system for viewing real-time information. https://plot.ly/python/getting-started/ https://plot.ly/python/

      • Jesse Boxell

        So, 6 hours of learning Python and I think I’m ready to tackle this code. 🙂

  • Thorwald Westmaas

    Very interesting.

    Now, the Xively folks seem to have gone completely corporate. Are there any alternatives for private people to log/visualize their Arduino-generated data output in the cloud ?

  • Segun Adebayo

    I my data is not posting to xively although csv file was successful. pls what could i be doing wrong

  • Ys Hero

    I want to do the same store a state of pushbutton from arduino to txt file using pyserial but i’ve problems first the Serial Monitor keep displaying ‘button off ‘button of or ‘button on ‘button on ‘& it doesnt write to my file the state of button could you please help here are my codes :
    int pushButton =2; //digital pin 2 has a push button attached to it. Give it an name
    int led=13;
    //the setup routine runs once when you press reset
    void setup() {
    pinMode(pushButton, INPUT); //make the push button’s pin an input
    pinMode(led,OUTPUT);
    Serial.begin(9600);
    }

    void loop() {
    boolean buttonState = digitalRead(pushButton); //read the input pin
    if (buttonState ==1) {
    digitalWrite(led,HIGH);
    Serial.println(“Button is ON”);
    }
    if (buttonState ==0) {
    digitalWrite(led,LOW);
    Serial.println(“Button is OFF”);
    }

    delay(20); //delay in between reads for stability

    }

    # -*- coding: cp1252 -*-
    import serial,io
    from datetime import datetime
    connected=False
    outfile=’C:UsersYassineDesktop.txt’
    port = “COM1”
    baud = 9600
    ser = serial.Serial(port, baud, timeout=1)
    sio = io.TextIOWrapper(io.BufferedRWPair(ser, ser, 1), encoding=’ascii’, newline =’r’)

    with open(outfile,’a’) as f:
    while ser.isOpen():
    datastring=sio.readline()
    print datastring
    print datetime.now()
    f.write(datetime.now().isoformat() +’t’+ datastring +’b’ )
    f.flush()

    while not connected:
    pass
    serin=ser.read()
    connected=True
    ser.close()