This post will describe a project I worked on a few months ago during my spare time. Looking back, I see how much my skill set has changed, and how I would have attempted this project in a completely different way. Nevertheless, this is the whole point of documenting one’s work – and one of the primary goals of my blog (see About section). It allows me to keep track of my progress (i.e. see how silly/naive I was before). Parts of the following post were actually written for a design journal which we had to keep for a class. This post is structured in a much more organized fashion, and is lengthy. I will revive and revamp this project next winter when I require my heater to turn on from a remote location (i.e. on my way home so that my room is warm), so keep an eye out.
Section I – Introduction
Section II – The Problem
Section III – Requirements & Constraints
Section IV – Analysis
Section V – Design
Section VI – Implementation
Section VII – Testing
Section VIII – Conclusion & Reflection
Section I – Introduction
The waterfall method, perhaps familiar to many of you, works very well for me. I will debrief at the end of this post to reflect on the effectiveness of the waterfall model and why I chose it over, say, the agile methodology.
Every single line of code, schematic, and instruction can be found on GitHub for you to refer to or use should you choose to replicate this project. Yes, this post is long to say the least. However, just to keep it short, I am not going to leave out important details – but please excuse any grammatical errors in my writing. Trust me, making everything work took me a lot longer than it did to write this up. For the record, I cannot claim full credit for this project. As a member of several vibrant online communities, I participate in the active and ongoing exchange of information, ideas and resources. The knowledge and experience of people I don’t even know has played an integral role in the success of this, and many of my projects, particularly when things simply do not seem to be working. This report is based on notes I have taken as I was working on the project.
On GitHub, you should find the following resources:
1. Circuit schematic (.fzz, .svg, and .png formats)
2. ATMega8 compatible code and makefile to use with a USBasp (easily adaptable)
3. Necessary header and library files to interface with the DHT-11
4. HTML and CSS code for frontend of webpage, along with necessary resources
5. Three PHP scripts (two for the webpage, one to interface with Android)
6. Android .apk application file (if you require source code, contact me)
7. Desktop program source code for reading RS232 port, and updating MySQL database (for use on Linux systems only, written in C)
8. Datasheet for DHT-11 temperature sensor
9. Datasheet for ATMega8 MCU
I look forward to your insight, criticism, and questions.
Section II – The Problem
My workspace contains a growing array of electrical, mechanical, and experimental equipment/tools which, on occasion, I need to monitor when I am away from home. Among other things, I need to know the temperature in my room or a potentially isolated experimental setting at any given. More importantly, I need to be able to access live data over the internet through a web browser, or mobile application capable of sending responses to the system from a remote location.
For example, I plan to use this device to activate/deactivate my heater to maintain a constant temperature in my room. I can also re-purpose this idea to ensure that I arrive to a warm environment and minimize energy wastage by keeping the heater off when I am away from home. That is a slightly separate project – albeit related and will utilize technology developed in this one – and will not be discussed in this post. However, I have to ensure my design is modular enough to adapt to this, and other, future applications of it. As I also mentioned, temperature data is just one variable I would like to monitor. I plan to monitor and induce environmental effects such as light (visible and infrared), vibration, and audible noise perturbations. By successfully implementing the system with temperature data first (almost as a proof of concept), my design should be able to accommodate and integrate these other factors into my system with relative ease.
Section III – Requirements & Constraints
What I need is a way to detect the temperature, and somehow retrieve it on my phone and/or a web browser within a budgetary constraint of $5. Here is a summary of the constraints:
• Accurately detect the temperature in my room (within one degree Celsius or less)
• Retrieve live data at any given time from anywhere in the world when the system is online
• Complete this project with parts exceeding no more than $5 (variable cost), less is highly preferred
• Things like cables, computers, etc. are not considered in the cost of the final product, as most people have such items already
• Be able to access data through a browser, and/or mobile phone application
• Be flexible enough to adapt to other similar systems (i.e. the core system should not be strictly for processing temperature data, and easily modifiable to accommodate other things)
You will notice that my requirements are relatively straightforward and simple. If you have ever participated in a design process yourself, then you probably know that requirements have a tendency to be deceivingly simple. Read on, and you will observe the amplified impacts of even the most seemingly minor roadblocks – on deadlines and cost in particular.
Section IV – Analysis
The requirements section laid out a very high level view of what needed to be done. Let us observe the task from a slightly more technical perspective (i.e. what intermediate steps may be required in order to reach the deployment stage). I will break down the problem into two parts:
1. Acquiring temperature data
• Tap into an off the shelf digital thermometer
• Don’t tap into the thermometer, but just place it at the desired location with a camera constantly pointing at it, and send pictures of the reading over the internet
• Use a temperature sensor with my own interface
• Use a thermistor (a resistor which varies in resistance based on ambient temperature)
2. Transmitting temperature data (assuming I have some way of acquiring it first)
• Send to local PC via USB, upload data to server, retrieve from server
• Send to local PC via RS232 (serial), upload data to server, retrieve from server
• Send to server directly over WiFi, retrieve from server
• Send to PC wirelessly, upload data to server, retrieve from server
- Radio transceiver (900 Mhz, 2.4 GHz, etc.)
So I have a few options, but I have to choose only one way to acquire the data, and one way to transmit it. My most significant constraints are accuracy, and cost, with complexity and time required to interface important factors to consider. It is important to note that all of these factors are relative, and not absolute in any sense. The tables below highlight each option to reflect the constraints and help me make a decision (green = good, yellow = moderate, red = poor):
I have experienced most of the above methods and have the resources to make them work, so I have a good idea with regards to what would be best for this project based on my aforementioned requirements. Another project with different requirements may justify using Bluetooth or USB instead of RS232. Anyway, the above charts make it pretty clear that a temperature sensor is the way to go, paired with serial communication to get the data onto the PC. Getting the data onto the Internet is the same in all cases.
The camera idea was ludicrous due to the intense data overhead, but I try not to reject even the silliest of ideas. All I really need is 4 bits of communication each time a temperature measurement is taken. Sending the lowest resolution photograph would easily take at least 200 kilobytes (400,000 times the data I really need). It would not even be data per se, just an image which I can’t do much analysis with.
Section V – Design
Now that the logistical details are out of the way, it is time for me to map out each and every technical detail which would lead to the deployment stage. My previously used high level approach is not enough for this stage of the product design process; the design stage of the waterfall method requires each and every detail to be laid out perfectly, because the next stage is implementation. If a mistake is made at this stage, the impact would be enormous – which is one drawback of the waterfall methodology. Here is a summary of the decisions I will have to make, and a justification for each decision. Keep in mind that all my decisions must link back to my requirements.
a) Choosing the temperature sensor
Starting with the temperature sensor, I ended up choosing the DHT-11 from Micropik (note that mine was not attached to a PCB as shown in the linked datasheet). There are a few reasons why I am going to choose this one:
• Cost: $0.99.
• Accuracy: It delivers TTL level digital signals to transmit its readings, with an accuracy of +/- 2 and a range of 0-50 degrees Celsius.
• Complexity: It uses the 1-wire communication protocol, complete with a checksum for added reliability. The 1-wire protocol is similar to I2C, for those who know what either are.
• Time: Estimated 6 hours to interface.
• Additional notes: Also reads humidity (potentially useful)
b) Choosing a MCU
Microcontrollers have really changed the way we deal with things on a regular basis. The first thing that pops into the minds of most people is the Arduino. I agree, the Arduino is a great tool. It is absolutely great for whipping up quick prototypes, and I own one for that reason. However, using an Arduino is not justified for this project based on my constraints, so I will go with an ATMega8-16PU standalone IC.
• Cost: At $1.35, you can’t beat it.
• Accuracy: N/A (any MCU would be more than sufficient in terms of accuracy for a simple project like this)
• Complexity: Atmel microcontrollers have a free C compiler, making programming them a cinch.
• Time: Included in the interfacing time above (6 hours).
• Additional notes: As you know, I am using RS232 to communicate with the PC. The ATMega8’s UART works at TTL levels. In other words, 0V is a logical 0, and 5V is a logical 1. The RS232 standard, on the other hand, demands voltage levels between +3V – +15V for logical LOW, and -3V – -15V for logical HIGH. Therefore, I need a level shifter to interface my MCU with RS232 standards. Read more about the RS232 standard here and TTL here.
c) Choosing a level shifter
There really isn’t a question when it comes to choosing a level shifter for an application like this. The MAX232 is famous for its presence in all TTL to RS232 (or vice versa) related applications, so why reinvent the wheel? You may ask, why not use two NAND gates with some transistors. Yes, those will do the job, but at the same time they will make my project more complex, costly, and time consuming (at higher baud rates, probably inaccurate too); thus, not matching my requirements. In the interest of completeness, the MAX232 cost me roughly $0.35.
d) Writing the program for the MCU (interfacing the sensor)
This will be discussed in grave detail in the next stage of the development process (implementation), but it’s quite simple, actually. If you look at the datasheet for the DHT-11 temperature sensor, you will notice (starting from section 5.1) that the entire communication process is described in precise detail. Simply put, it describes how my MCU and the sensor will initiate a conversation when I want the sensor to take its measurements, and how I need to interpret what it tells me to satisfy my purpose. To explain this concept more clearly, the next section will show you some pictures of what the signal looks like on an oscilloscope. Here is a summary of what the conversation looks like (please note that the signal line is pulled up to 5V by default via a pull up resistor):
• Initiating the conversation requires me (the MCU) to pull the signal line to the DHT-11 down from the HIGH to LOW for at least 18ms, and then pull it up for 20-40μs and wait for the sensor to respond. At this point, I need to switch the pin mode on my MCU from output to input.
• The DHT-11 should respond by sending an 80μs response by pulling the line to LOW, and confirming its response by bringing it back up to HIGH for 80μs – now indicating that it is ready to send data.
• Now, every bit of data begins with a 50μs LOW signal, followed by a signal representing a logical 1 or 0. If the DHT-11 pulls up the voltage for 26-28μs, then I know it’s a 0. Likewise, if the voltage is pulled up for 70μs, I know it’s a 1.
• The complete transmission is 40 bits in length: 8bit integral RH data + 8bit decimal RH data + 8bit integral T data + 8bit decimal T data + 8bit check sum (RH is humidity, T is temperature). I am interested in the third data packet.
• When the last bit is transmitter, the DHT-11 will send a 50μs LOW response, and then let the signal go for the pull up resistor to take over. This action indicates that the conversation is complete.
The next part of the code for the MCU will transmit the data received from the sensor to the UART (universal asynchronous receiver transmitter) built into the MCU. The output from the ATMega8 will go into the MAX232 to be level shifted, and then to the PC. Because communication is asynchronous, among other things, I will have to choose a specific baud rate for communication.
e) Creating the circuit
Now that I know what components will comprise the hardware side of this project, I need to create a schematic. Initially, I plan to leave the circuit on a solderless breadboard. I usually leave my projects on solderless breadboards for a while so that any modifications can be easily made, before I opt for PCB etching or soldering the circuit onto a perforated board. I probably will not have time for that until the winter break anyway.
Here is what the schematic looks like (see below). It is very straightforward, and there is not much going on. I will likely be drawing power from a USB port, but if not, I can replace the battery shown with a 78L05 voltage regulating circuit.
Writing a program to capture incoming data, and send it to a server
As described earlier, from the hardware side, a level shifter is required, and I chose a MAX232 IC. The MCU would have also been programmed to send the information it receives from the DHT-11 over its UART by this point. Another C program will have to be written which will capture the data coming from the MAX232 (read: MCU), and post this instantly to a remotely hosted database.
Before moving forward, it came to my realization that my computer does not have a serial communication port – unfortunately, only USB. I utilized an RS232 to USB adapter cable, based on the Prolific Pl2303 serial to USB bridge controller. Although I’m getting ahead of myself, I will eventually describe how I discovered my chip was counterfeit, and was the cause of driver issues which, in turn, resulted in my project’s progress being delayed by two whole days.
The program should implement several libraries responsible for opening the serial COM port (file found under /dev/ttyUSB0 of the Linux file system), and several other libraries required to access and send queries to a MySQL database. The libraries include: my_global, mysql, fcntl, errno, termios, and the regular old stdio and string. I also have to ensure that the MySql-C development package was installed in my computer in order to access those libraries. For the record, I am running this on a Linux machine.
f) Organizing the data on the server
As far as my experience with these kinds of projects goes, the only apparent and simple option which also seemed appropriate for this task was to utilize a database. SQL is a very well known, easy to use, and a light tool for data basing, so I will choose to go with it. I am accustomed to the LAMP stack (Linux, Apache, MySQL, PHP), which should make the design of the data management system relatively easy. In order to store the data in my database in its simplest form, I will require the latest temperature reading, and a time stamp. This data, of course, can easily be re-purposed to deliver weekly/daily averages, alerts, etc.
The data will be stored in a database called roomtemp, with two integral columns: date, and temperature. Each will be of type varchar. The aforementioned program running on my local PC will constantly read data coming from my MCU over the RS232 port, and update the fields in the database. I will be using a remote server I own in order to send, store, and retrieve data.
g) Displaying the data
Initially, I chose to display the data on a website. I will continue, but have also decided to create a simple mobile application capable of retrieving the data. Both work on the same precept, but just require different languages to implement.
• The website will be simple, based on an HTML and CSS frontend, and PHP and AJAX backend.
• The mobile application will be based on Android, and coded in Java and XML.
• It, similar to the PHP code, will access the SQL database, and retrieve the necessary information.
• I will use the Android SDK to achieve this, and will utilize a MySQL library.
The ideal display for both the website and mobile application will include the temperature, date of the measurement, and whether the system is online or offline. Although there are several ways to evaluate whether the system (i.e. the physical hardware in my room) is connected to the internet, I will choose to compare the time stamp of the last measurement to that of the current time of viewing. If more than a minute has elapsed since a value has been uploaded to the database, I will assume the system has gone offline. The instant a new value is sent to the database, the PHP code will declare the system as online (for at least one minute, as per the previously described procedure).
To keep things simple, the CSS and HTML frontend will consist of a background and centered area functioning as the HUD (as described above). Although not a constraint, the interface should look good. Finally, I will host this project at: www.treehouseprojects.ca/roomtemp (I haven’t turned the system on for a while now).
Section VI – Implementation
This section will be divided in two parts. The first part outlines, with evidence, all the steps I followed in order to achieve the final results. The second, more interesting part, will discuss the problems I faced, and how they impacted my requirements and constraints.
Part A – Putting It All Together
a) As shown in the Section V(e) of this report, I created a circuit schematic which I would follow in order to complete the hardware side of the system. Putting the circuit together was extremely easy because I spent a lot of time on the design. I installed the circuit on a solderless breadboard, but will migrate it over to a PCB in due course. The solderless platform makes prototyping, testing, and modifications very easy. Below, you will find a picture of my hardware, with the integral components labelled:
Interfacing the Sensor
b) Section V(d) of this report, and the datasheet for the DHT-11 temperature sensor, outlines in exact detail the procedure for communication between the sensor and the MCU. In order to understand this process better, and provide the reader of this report with a visual representation of the communication protocol in action, I hooked up the sensor to an oscilloscope. After fiddling around with various trigger settings and resolutions, I came up with the following result for the 8 bit data packet containing the temperature data:
Note (aside from the terrible glare) that this packet of data is sent in roughly 0.000735 seconds, which made sense considering each high bit lasts 70μs, whereas each low one lasts roughly 25μs (both separated by the signal being held low for 50μs). As shown in the image, that packet of data corresponds to the binary value of 0b00010111, and in base 10 decimal it translates to 23 (i.e. 23 degrees centigrade). That meant that the lab was at a comfortable 23 degrees at the time of this sample. Having confirmed that the sensor was following the correct communication protocol, I simply programmed my MCU to capture the above bits. During this processes, I found someone had also used this sensor with a different MCU, so I adapted parts of his code into my own (confirming that his code was open source).
Interfacing the MCU with my Computer
c) Originally, I planned to run this entire project on a Windows machine. Due to some unexpected problems with my RS232-USB cable (explained later in this section), along with some oddities when compiling the MySQL libraries, I found switching entirely to my Linux system would speed the process up quite a bit. The code simply opens the serial port (located at /dev/ttyUSBx), as well as a connection to my server, and continuously reads the port for any new data coming in from the MCU. Once it finds any new data (approximately once a second), it processes it, integrates it into a MySQL query, and updates the database via the query.
Displaying the Data
d) As planned, I went through with creating a web interface for this project, as well as a mobile application. Both do essentially the same thing, however, the reason I also chose to create the mobile application is so that I have a platform to interact with when I expand the capability of this system. For example, if I want to turn a mechanism on or off based on a temperature reading, a mobile application will be a secure way of doing so. The web page (www.treehouseprojects.ca/roomtemp) was built with a CSS and HTML front end as decided in Section V, and a PHP backend which communicates which communicates with my server via scripting. The Android application did not use any server related libraries (explained in the next part of this section), and instead made calls to another script on my server and retrieved the necessary information over an HTTP connection. The webpage may be viewed by visiting www.treehouseprojects.ca/roomtemp, and the app may be installed on your phone by installing the attached APK file. Some images of the application are shown below (sorry about the glare on my phone’s screen):
Both the app and the website update once a second when the system in my room is online, and they both display a time stamp of the latest update to the database corresponding to the temperature reading being displayed. The webpage also shows whether the system is online or not, however this feature has yet to be implemented on the mobile application.
Part B – The Problems
Part A made everything look easy and straightforward, and for many of you, a project like this may be very simple. I initially thought I could complete it within two weeks (working 1-2 hours a day), but it became apparent that every step of the process would introduce challenges which I would had never anticipated, often throwing me off schedule. In hindsight, surpassing technical hurdles has always been a satisfying and tremendous learning experience. After working on several projects under the guidance of the waterfall method, I have learned that spending as much time on the design stage as possible will make the project move much faster with far fewer hiccups. Although there are way too many problems
which I ran into throughout this project to describe here, I will stick to a few key ones which negatively impacted my requirements most.
1) If you are meticulous enough to notice that the ATMega8 has a default clock speed of 1MHz, you may have suspected that capturing data with microsecond precision may have been problematic at such low speeds. I thought I may have been able to get away with the internal clock, but realized the hard way that the internal clock is simply too imprecise (even if I choose a faster speed). I was often getting garbled data in my terminal, and I narrowed down its cause to two potential culprits. First, I may have been communicating with the sensor incorrectly. Second, the UART may not be configured correctly. It was fairly simple to identify the culprit as miscommunication, by testing the UART through other means. I described the symptoms of this problem to some of pals online, and they were quick to point out the obvious. I immediately set the fuse bits and configured my MCU to run off a 16MHz external crystal, and it was smooth sailing from there.
Impact: Project delayed by 5 hours
2) The main reason I switched to a Linux distro for this project was because my RS232-USB adapter was not functioning correctly. No drivers were working correctly with the device. I later found out that the PL2303 chip utilized by the adapter was often sold in the counterfeit products market. Having purchased the adapter online from Hong Kong, it seemed likely that this was the case. Luckily, Linux was not as fussy. Although I did end up solving the problem on Windows as well, configuring serial communication on my Linux box was also much easier. Impact: Project delayed by 8 hours
IP address issues
1) This was the first time I was connecting to a remote (non localhost) server at my new hosting company, and ran into a key problem. The company’s hosting plan involves remote SQL connections, but only from specified IP addresses. Initially, I added my computer’s IP address, and everything worked great. The problems arose when I realized public IP addresses are dynamic, and that would pose a problem with connectivity. Not only would I have to add a new IP address to the list every time it changed, but this meant no one else would be able to access my database when they visited my website either (because everyone has a different IP). I contacted the company about this issue, and they were not able to resolve the situation. I could not accept this, and realized that if I enter a wildcard instead of a digit, then any IP address could access my database. For example, instead of allowing 127.468.21.7, I could allow %%%.%%%.%%.% to access my database. To my pleasant surprise, this worked!
Impact: Project delayed by 6 hours
Serial buffer overflows
2) As I learned the hard way, whenever you connect a device (like a serial one), all of its output is stored in a buffer. Often times, my hardware would be connected to the PC, still sending data, but I wasn’t retrieving it. In other words, because I did not open the port, the data would just accumulate there. I scoured the web to figure out how I could flush the data before reading it, but nothing seemed to work. After even more searching, I found out by sheer happenstance that the thread needs to sleep a while after the flush in order for it to work. No one really knew why, but it worked. These are the kind of problems I really dislike, because I did not learn much and cannot clearly explain my solution.
Impact: Project delayed by 6 hours
Android SQL connectivity
3) I love Android, but never did I expect to find that there are no libraries available for use to connect with an SQL database. I was really counting on this, and my lack of research during the design stage resulted in an unpleasant surprise. Not surprisingly, many other people around the world have a similar requirement, and the best solution I found was the following. I had to create a PHP script on my server which would act as the link between my app and the database. The Android app would make an HTTP call with an input stream on the remotely hosted script, and would capture whatever the script would return. I would then process the data from the input stream’s buffer. Many people choose to use JSON in order to make things more efficient because PHP can output in JSON format, and Android can parse it. I decided to skip that route because I only needed to retrieve two values.
Impact: Project delayed by 4 hours
4) Although not exactly a problem, I am stickler for good design. The problem is that I am terrible at design, and it takes me a long time to like something. I spent hours trying to visualize patterns and colors, trying to understand what I really wanted. Some people may not like the design, but I am satisfied with it. In my opinion, poor designs gives someone the impression that I am not serious about all aspects of my work, and detracts hugely from the professional appeal of it.
Impact: Project delayed by 2 hours
Evidently, all my problems were fixed, but also severely impacted the timeliness of the project’s completion. With only 1-2 hours a day to work on this (more on weekends), every extra minute spent debugging or fixing issues really hurt. Bear in mind that the issues described above are just the major ones. I must have spent another 12 hours resolving smaller issues. I attribute this to my poor design and lack of experience, obviously unacceptable excuses in the real world.
Section VII – Testing
I apologize, but I do not have much to write here. I have been testing the system for about one week, and it seems to be working flawlessly!
One minor issue, however, is browser compatibility. I received a report from one of the people I asked to test my system that the web interface does not work on an old version of Internet Explorer. Honestly, I have not used IE since the Stone Age, but I identified the problem to be attributed to the version of CSS. Old browsers (namely IE) do not support CSS3, which is what I used for the frontend. You may notice the border radius and opacity of the box containing the data; well, those are new features of CSS3 – ones which I really like. For a real product, I would not leave any stones unturned, and would ensure that my product works on every single platform.
Section VIII – Conclusion and Reflection
I always keep a log of my work in writing to gauge my progress. I went into this project with an idea, and materializing any idea is a gruelling, but rewarding process. This project is far from complete, but I have built a foundation for several applications of this device which I have in mind (as described in Sections I and II). I would like to to evaluate how successful this project was by comparing my results against the requirements.
Overall, I did meet most of my requirements. I was able to accurately detect my room temperature, retrieve data at any time from any place, complete the project within a budgetary constraint of $5 (~$4.25), create a mobile application and web interface, and maintain flexibility/scalability of the system in order to accommodate future expansions. I missed my mark on the timeliness of this project though, by one whole week. This comes as no surprise considering all the problems I ran into (as described in the previous section) impacted only the timely completion of my project. I am sure many of you can attest to the fact that completely unexpected problems have a tendency of arising at the worst times.
There are two solutions I have found for this problem in relation to the waterfall model. The first is to design, design again, and design once more. My seniors have always told me that if I spend 90-95% of my time at the design stage, I will hit my mark 99% of the time. Most people like me are always racing to hit the implementation stage of a project, but we pay the price later. The second solution has to do with working in a team. In retrospect, many setbacks were actually extremely easy to remedy, but a one man team leads to tunnelled vision and cognitive anchors. Engineering has been, and will always be a team sport.
This is actually a great segue into my reasons for choosing the waterfall method. I like the waterfall method because it is very sequential, well defined, and perfect for projects handled by one person. Iterative methods, like the agile methodology, work great when you work in teams where many people can keep contributing and reworking their share of the work until everything integrates and works seamlessly. When you work alone, as I did in the case of this project, you can usually only work on one thing at a time, so a sequential chain of events greatly facilitates this process.
The simplicity of the waterfall method has the added advantage of easier logging. In the real world, often times we have to pick up and complete someone else’s project. If the previous owner of the project was following an iterative approach, finding your bearings may prove to be an extremely difficult undertaking. Under the waterfall method, you will know exactly where you are taking over. The key disadvantage, however, is that due to the linearity of the waterfall method, moving backwards is hard – which is why I emphasized the vitality of the design stage. Really, the waterfall method is a double edge sword, and is obviously not appropriate or most effective in every setting.
Thank you for taking the time to read through this report. As I stated at the beginning, feedback is extremely important to me, so please let me know if you have any questions, comments, or suggestions. Feel free to use any of the files on GitHub for your own purposes, but do note that I have omitted my server’s access credentials (so you will need your own).