[Solved] Python, pySerial and missing data

This is the place for queries that don't fit in any of the other categories.

[Solved] Python, pySerial and missing data

Postby pe2bz » Wed Jul 24, 2013 4:57 am

Hi,
Perhaps I am not at the right place with my question but I have no idea where to ask so I am going to start over here:

I am running a python script to collect data from a Smart Meter to create graphs from my power use and solar delivery. All is running on a Raspberry Pi, which is connected with an USB cable to the Smart Meter. I am running a script which collects the data at given times (every 5 minutes, started as cron job) and creates a text file (running the python script and copy it's output to the textfile) where rrdtools gets it's data from to create a graph.

Sometimes all runs well for a day, sometimes I have several holes in my graph because the collected data does not fit the need for the rrdtool. I changed the script to save each collected file from running the python script with a filename containing date and time to be able to observe the differences. See the two examples below:

A good file:

Code: Select all
('DSMR P1 uitlezer', '1.0')
Control-C om te stoppen
daldag      01782
piekdag     01835
dalterug    00000
piekterug   00007
afgenomen vermogen       500  W
teruggeleverd vermogen   0  W
Gas                      2070889  dm3

A bad file:

Code: Select all
('DSMR P1 uitlezer', '1.0')
Control-C om te stoppen
daldag      01782
piekdag     01835
piekterug   00007
afgenomen vermogen       460  W
teruggeleverd vermogen   0  W
daldag      01782
piekdag     01835
piekterug   00007
afgenomen vermogen       450  W


I can run the python script manually and then also it goes about 50 percent of the time wrong.

I did start Minicom and had it capture the same data (/dev/ttyUSB0 ) for about one hour. The Smart Meter outputs it's data (push) every 10 seconds so I had a capture file with more than 320 readings, which where all 100 percent OK. So I did install the latest pySerial and updated Python to the 3.3.2 version. After restarting the Raspberry Pi it did run ok for some hours but again I am getting error readings and holes in my graph.

I will paste the script and I hope someone can give me an idea what could be the problem (and solution) with my readings. I did not create the python script myself, it was available online for use.

Thanks for your time,
Ben

Code: Select all
#
# DSMR P1 uitlezer
# (c) 10-2012 - GJ - gratis te kopieren en te plakken

versie = "1.0"
import sys
import serial

################
#Error display #
################
def show_error():
    ft = sys.exc_info()[0]
    fv = sys.exc_info()[1]
    print("Fout type: %s" % ft )
    print("Fout waarde: %s" % fv )
    return


################################################################################                                                  ################################################################
#Main program
################################################################################                                                  ################################################################
print ("DSMR P1 uitlezer",  versie)
print ("Control-C om te stoppen")

#Set COM port config
ser = serial.Serial()
ser.baudrate = 9600
ser.bytesize=serial.SEVENBITS
ser.parity=serial.PARITY_EVEN
ser.stopbits=serial.STOPBITS_ONE
ser.xonxoff=0
ser.rtscts=0
ser.timeout=20
ser.port="/dev/ttyUSB0"

#Open COM port
try:
    ser.open()
except:
    sys.exit ("Fout bij het openen van %s. Programma afgebroken."  % ser.name)                                                   


#Initialize
# stack is mijn list met de 20 regeltjes.
p1_teller=0
stack=[]

while p1_teller < 20:
    p1_line=''
#Read 1 line
    try:
        p1_raw = ser.readline()
    except:
        sys.exit ("Seriele poort %s kan niet gelezen worden. Programma afgebroke                                                  n." % ser.name )
    p1_str=str(p1_raw)
    #p1_str=str(p1_raw, "utf-8")
    p1_line=p1_str.strip()
    stack.append(p1_line)
# als je alles wil zien moet je de volgende line uncommenten
#    print (p1_line)
    p1_teller = p1_teller +1

#Initialize
# stack_teller is mijn tellertje voor de 20 weer door te lopen. Waarschijnlijk m                                                  ag ik die p1_teller ook gebruiken
stack_teller=0

while stack_teller < 20:
   if stack[stack_teller][0:9] == "1-0:1.8.1":
        print "daldag     ", stack[stack_teller][10:15]
   elif stack[stack_teller][0:9] == "1-0:1.8.2":
        print "piekdag    ", stack[stack_teller][10:15]
# Daltarief, teruggeleverd vermogen 1-0:2.8.1
   elif stack[stack_teller][0:9] == "1-0:2.8.1":
        print "dalterug   ", stack[stack_teller][10:15]
# Piek tarief, teruggeleverd vermogen 1-0:2.8.2
   elif stack[stack_teller][0:9] == "1-0:2.8.2":
        print "piekterug  ", stack[stack_teller][10:15]
# Huidige stroomafname: 1-0:1.7.0
   elif stack[stack_teller][0:9] == "1-0:1.7.0":
        print "afgenomen vermogen      ", int(float(stack[stack_teller][10:17])*                                                  1000), " W"
# Huidig teruggeleverd vermogen: 1-0:1.7.0
   elif stack[stack_teller][0:9] == "1-0:2.7.0":
        print "teruggeleverd vermogen  ", int(float(stack[stack_teller][10:17])*                                                  1000), " W"
# Gasmeter: 0-1:24.3.0
   elif stack[stack_teller][0:10] == "0-1:24.3.0":
        print "Gas                     ", int(float(stack[stack_teller+1][1:10])                                                  *1000), " dm3"
   else:
        pass
   stack_teller = stack_teller +1

#print (stack, "\n")

#Close port and show status
try:
    ser.close()
except:
    sys.exit ("Oops %s. Programma afgebroken." % ser.name )



And here is the script that runs the python script and creates the text files:

Code: Select all
#!/bin/bash
NOW=$(date +"%Y%m%d%H%M%S")

sleep 3
# 1 keer per 5 minuten, uitlezen via seriele interface van de slimme meter
su - raspbian -c "python /home/raspbian/P1uitlezer.py > /home/raspbian/P1uitlezer.txt" 2>&1 >/dev/null

su - raspbian -c "python /home/raspbian/P1uitlezer.py > /home/raspbian/P1uitlezer-$NOW.txt" 2>&1 >/dev/null
sleep 3
#####################################################################################
# Bijwerken stroomverbruik
#####################################################################################
powerin=`/bin/grep afgenomen /home/raspbian/P1uitlezer.txt | /usr/bin/awk '{print $3}'`
powerout=-`/bin/grep teruggeleverd /home/raspbian/P1uitlezer.txt | /usr/bin/awk '{print $3}'`
/usr/bin/rrdtool update /var/www/rrdpower/power.rrd N:$powerin:$powerout

DIR="/var/www/rrdpower"
INPOWER_COLOR="#B5B690"
OTPOWER_COLOR="#3B3131"
rrdtool graph $DIR/power_hourly.png --start -1h DEF:powerin=$DIR/power.rrd:powerin:AVERAGE LINE1:powerin$INPOWER_COLOR:"Ingaand vermogen [W]" \
DEF:powerout=$DIR/power.rrd:powerout:AVERAGE LINE1:powerout$OTPOWER_COLOR:"Uitgaand vermogen [W]"
rrdtool graph $DIR/power_daily.png --start -1d DEF:powerin=$DIR/power.rrd:powerin:AVERAGE LINE1:powerin$INPOWER_COLOR:"Ingaand vermogen [W]" \
DEF:powerout=$DIR/power.rrd:powerout:AVERAGE LINE1:powerout$OTPOWER_COLOR:"Uitgaand vermogen [W]"
rrdtool graph $DIR/power_weekly.png --start -1w DEF:powerin=$DIR/power.rrd:powerin:AVERAGE LINE1:powerin$INPOWER_COLOR:"Ingaand vermogen [W]" \
DEF:powerout=$DIR/power.rrd:powerout:AVERAGE LINE1:powerout$OTPOWER_COLOR:"Uitgaand vermogen [W]"
rrdtool graph $DIR/power_monthly.png --start -1m DEF:powerin=$DIR/power.rrd:powerin:AVERAGE LINE1:powerin$INPOWER_COLOR:"Ingaand vermogen [W]" \
DEF:powerout=$DIR/power.rrd:powerout:AVERAGE LINE1:powerout$OTPOWER_COLOR:"Uitgaand vermogen [W]"
rrdtool graph $DIR/power_yearly.png --start -1y DEF:powerin=$DIR/power.rrd:powerin:AVERAGE LINE1:powerin$INPOWER_COLOR:"Ingaand vermogen [W]" \
DEF:powerout=$DIR/power.rrd:powerout:AVERAGE LINE1:powerout$OTPOWER_COLOR:"Uitgaand vermogen [W]"
Last edited by pe2bz on Fri Jul 26, 2013 3:32 pm, edited 2 times in total.
pe2bz
 
Posts: 3
Joined: Tue Jul 23, 2013 5:41 pm

Re: Python, pySerial and missing data

Postby metulburr » Wed Jul 24, 2013 7:38 am

New Users, Read This
OS Ubuntu 14.04, Arch Linux, Gentoo, Windows 7/8
https://github.com/metulburr
steam
User avatar
metulburr
 
Posts: 1509
Joined: Thu Feb 07, 2013 4:47 pm
Location: Elmira, NY

Re: Python, pySerial and missing data

Postby pe2bz » Wed Jul 24, 2013 5:23 pm

metulburr wrote:use code tags


Sorry, my appologies. I completely missed that.

@Yoriz : Thanks for adding the code tags for me !

Kind regards,

Ben
pe2bz
 
Posts: 3
Joined: Tue Jul 23, 2013 5:41 pm

Re: Python, pySerial and missing data

Postby pe2bz » Fri Jul 26, 2013 3:31 pm

It seems that the problem is solved.

It's running for 24 hours now with no single error. I changed the xonxoff setting in the P1uitlezer.py from "0" to "1" and since then no more errors appeared.

Code: Select all
#Set COM port config
ser = serial.Serial()
ser.baudrate = 9600
ser.bytesize=serial.SEVENBITS
ser.parity=serial.PARITY_EVEN
ser.stopbits=serial.STOPBITS_ONE
ser.xonxoff=1
ser.rtscts=0
ser.timeout=20
ser.port="/dev/ttyUSB0"


Thanks all,

Kind regards,
Ben
pe2bz
 
Posts: 3
Joined: Tue Jul 23, 2013 5:41 pm


Return to General Coding Help

Who is online

Users browsing this forum: Google [Bot], W3C [Linkcheck] and 3 guests