Saturday, April 29, 2017

Github updated

I have started to collect some useful tool source code for Arduino projects into a github repository.  Some of these projects have been published on this blog, but not all of them.  I will try to to maintain anything that I post here also on github for the convenience of those that find any of my examples useful.

You can find my Arduino github repository here.

Thursday, April 20, 2017

UPDATED: A Silly Little Project

I have a Freetronics EtherTen Arduino Uno-compatible board that includes SD card and Ethernet support on-board.  It is a cool little board that I use a lot for prototyping a lot of projects.  I also have a Freetronics LCD and Keypad Shield that is usually attached to it.  This combo usually knocks around in my go bag, but in-between I wanted it to provide some useful function at my workspace when not otherwise occupied.

What I decided to do is build a two time zone clock that keeps synchronized to the NIST time servers.



So, the gist here was to have UTC and my local time zone displayed along with the local time zone date.  I didn't ever want to have to mess with setting it and wanted it to automatically handle daylight savings time.  The clock uses the millisecond timer to keep time, resetting to time.nist.gov every hour.  The update from the time server takes a couple seconds to accomplish, so we don't want to hammer on the network.  Once per hour appears to be completely adequate.

The code is a collection of code snips from the internet and some clever logic to avoid a bunch of if-then-else logic when calculating daylight savings time.  I can't really claim much ownership of anything other than this unique instance of these snippits.  My thanks goes out to the original authors for the inspiration and willingness to share code.

So, let's walk through the code from the top.  Here we have all necessary include files and initialization of the LCD and Ethernet bits.  We use UDP to communicate with the NIST server.

// UDP NTP Client - Implements an NTP set clock
//
// Displays UTC and Pacific time and date on 16x2 LCD display synchronized to internet time server time.nist.gov.
// Calculates daylights savings time, handles leap years
//
// Jeff Whitlatch ko7m - 23 Mar 2017

#include <Ethernet.h>
#include <EthernetUdp.h>
#include <LiquidCrystal.h>

// Freetronics 16x2 version
LiquidCrystal lcd(8,9,4,5,6,7);

// MAC address to use for Ethernet adapter
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };

unsigned int localPort = 8888;       // local port to listen for UDP packets

char timeServer[] = "time.nist.gov"; // time.nist.gov NTP server

const int NTP_PACKET_SIZE = 48; // NTP time stamp is in the first 48 bytes of the message

byte packetBuffer[ NTP_PACKET_SIZE]; //buffer to hold incoming and outgoing packets

// A UDP instance to let us send and receive packets over UDP
EthernetUDP Udp;

Here we set up a textual month array and process the setup() function which sets up the serial debug port, ethernet port and starts the UDP listener on the local port 8888.

const char *szMonth[] = { "Jan", "Feb", "Mar", "Apr", "May", "Jun",
                          "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" };

void setup()
{
  // Open serial communications and wait for port to open:
  Serial.begin(115200);
  Serial.println("Setting up ethernet");

  // Init the LCD display
  lcd.begin(16, 2);
  
  // start Ethernet and UDP
  if (Ethernet.begin(mac) == 0)
  {
    Serial.println("Failed to configure Ethernet using DHCP");
    // no point in carrying on, so do nothing forevermore:
    for (;;);
  }
  Udp.begin(localPort);
}

One-time initialization of some time-related globals is next.

unsigned long secsSince1900 = 0;

int hour = 0;
int minute = 59;
int second = 58;
int day = 0;
int month = 0;
int year = 1900;
int dow = 0;
boolean fDST = false;

char buf[256];

Now, we get into the main loop which keeps the display updated predominantly.  Some optimization should be done here to only update the LCD when it changes, but in this simple code, I am updating it every time through the main loop.

Once per hour and on first boot, we send a NTP time request packet to the NIST server and parse the result that is returned.  We just extract the information of interest.  NTP is described in RFC 1305 and returns time as the number of seconds since 1 Jan 1900.

void loop()
{
  // Once every hour, update from the net, this takes a couple seconds so we don't do every time thru loop
  if (minute == 59 && second == 58)
  {
    sendNTPpacket(timeServer); // send an NTP packet to a time server
  
    delay(1000);
  
    if (Udp.parsePacket())
    {
      // We've received a packet, read the data from it
      Udp.read(packetBuffer, NTP_PACKET_SIZE); // read the packet into the buffer
  
      // the timestamp starts at byte 40 of the received packet and is four bytes,
      // or two words, long. First, extract the two words:
  
      unsigned long highWord = word(packetBuffer[40], packetBuffer[41]);
      unsigned long lowWord = word(packetBuffer[42], packetBuffer[43]);
      // combine the four bytes (two words) into a long integer
      // this is NTP time (seconds since Jan 1 1900):
      secsSince1900 = highWord << 16 | lowWord;
      Serial.print("Seconds since Jan 1 1900 = ");
      Serial.println(secsSince1900);
    }
  }

Now Unix time is the number of seconds since 1 Jan 1970, so we need to convert.

  // Unix time starts on Jan 1 1970. In seconds, that's 2208988800:
  const unsigned long seventyYears = 2208988800UL;
  
  // subtract seventy years:
  unsigned long epoch = secsSince1900 - seventyYears;

Now, we can calculate the time of day.

  hour = (epoch / 60 / 60) % 24;
  minute = (epoch / 60) % 60;
  second = epoch % 60;

The algorithm implements a proleptic Gregorian calendar. That is, the rules which adopted the Julian calendar in 1582 in Rome are applied both backwards and forwards in time. This includes a year 0, and then negative years before that, all following the rules for the Gregorian calendar.  The accuracy of the algorithms under these rules is exact, until overflow occurs. Using 32 bit arithmetic, overflow occurs approximately at +/- 5.8 million years. Using 64 bit arithmetic overflow occurs far beyond +/- the age of the universe. The intent is to make range checking superfluous.


These algorithm internally assumes that March 1 is the first day of the year. This is convenient because it puts the leap day, Feb. 29 as the last day of the year, or actually the preceding year. That is, Feb. 15, 2000, is considered by this algorithm to be the 15th day of the last month of the year 1999. This detail is only important for understanding how the algorithm works.

Additionally the algorithm makes use of the concept of an era. This concept is very handy in creating an algorithm that is valid over extremely large ranges. An era is a 400 year period. As it turns out, the calendar exactly repeats itself every 400 years. And so we first compute the era of a year/month/day triple, or the era of a serial date, and then factor the era out of the computation. The rest of the computation centers on concepts such as:

  • What is the year of the era (yoe)? This is always in the range [0, 399].
  • What is the day of the era (doe)? This is always in the range [0, 146096].

Further details on the calculations can be obtained from the wonderful write-up here.

  // Algorithm: http://howardhinnant.github.io/date_algorithms.html#civil_from_days
  // an era is a 400 year period starting 1 Mar 0000.  
  long z = epoch / 86400 + 719468;
  long era = (z >= 0 ? z : z - 146096) / 146097;
  unsigned long doe = static_cast<unsigned long>(z - era * 146097);
  unsigned long yoe = (doe - doe/1460 + doe/36524 - doe/146096) / 365;
  unsigned long doy = doe - (365*yoe + yoe/4 - yoe/100);
  unsigned long mp = (5*doy + 2)/153;
  
  month = mp + (mp < 10 ? 3 : -9);
  day = doy - (153*mp+2)/5 + 1;
  year = static_cast<int>(yoe) + era * 400;
  year += (month <= 2);

  dow = dowFromDate(day, month, year);
  fDST = isDST(day, month, dow);

Now, we have all the bits necessary for printing, so we build a text buffer and send it to the LCD display.

  // Build the print buffer
  sprintf(buf, "%2d:%02d UTC %2d %s", hour, minute, day, szMonth[month-1]);
  lcd.setCursor(0, 0);
  lcd.print(buf);

  // Print the second time zone
  if (hour < 8) hour += 24;
  sprintf(buf, "%2d:%02d %s  %d", hour - 7, minute, fDST ? "PDT" : "PST", year);
  lcd.setCursor(0, 1);
  lcd.print(buf);
  
  // Keep the network alive every 60 seconds
  if (second % 60 == 0)
      Ethernet.maintain();
      
  // Accuracy depends on system clock accuracy, corrected once an hour.
  delay(1000);
  secsSince1900++;
}

I am not going to detail the NTP protocol here.  The following is sufficient for the meager needs of this simple application.  We build the UDP request packet and send if off to time.nist.gov.

// send an NTP request to the time server at the given address
void sendNTPpacket(char* address)
{
  // set all bytes in the buffer to 0
  memset(packetBuffer, 0, NTP_PACKET_SIZE);
  // Initialize values needed to form NTP request
  // (see URL above for details on the packets)
  packetBuffer[0] = 0b11100011;   // LI, Version, Mode
  packetBuffer[1] = 0;     // Stratum, or type of clock
  packetBuffer[2] = 6;     // Polling Interval
  packetBuffer[3] = 0xEC;  // Peer Clock Precision
  // 8 bytes of zero for Root Delay & Root Dispersion
  packetBuffer[12]  = 49;
  packetBuffer[13]  = 0x4E;
  packetBuffer[14]  = 49;
  packetBuffer[15]  = 52;

  // all NTP fields have been given values, now
  // you can send a packet requesting a timestamp:
  Udp.beginPacket(address, 123); //NTP requests are to port 123
  Udp.write(packetBuffer, NTP_PACKET_SIZE);
  Udp.endPacket();
}

The determination of daylight savings time is a bit of a moving target over time, but this function encapsulates the yes/no decision to allow for appropriate correction of the local time display along with the date.

// In most of the US, DST starts on the second Sunday of March and 
// ends on the first Sunday of November at 2 AM both times.
bool isDST(int day, int month, int dow)
{
  if (month < 3 || month > 11) return false;
  if (month > 3 && month < 11) return true;

  int prevSunday = day - dow;

  // In March, we are DST if our previous Sunday was on or after the 8th.
  if (month == 3) return prevSunday >= 8;

  // In November, we must be before the first Sunday to be DST which means the
  // previous Sunday must be before the 1st.
  return prevSunday <=0;
}
  This one is kind-of fun.  See if you can describe this one liner in words...  :) 
  // Day of week calculation
int dowFromDate(int d, int m, int y)
{
  return (d += m < 3 ? y-- : y - 2, 23*m/9 + d + 4 + y/4- y/100 + y/400)%7;

}

And, that is really all there is to it.  I am sure the reader can adapt the code for different time zone pairs, but if you run into difficulty I will be happy to assist if you drop me a note at ko7m at ARRL dot net.  For your convenience, here is the entire code file:

// UDP NTP Client - Implements an NTP set clock
//
// Displays UTC and Pacific time and date on 16x2 LCD display synchronized to internet time server time.nist.gov.
// Calculates daylights savings time, handles leap years
//
// Jeff Whitlatch - 23 Mar 2017

#include <Ethernet.h>
#include <EthernetUdp.h>
#include <LiquidCrystal.h>

// Freetronics 16x2 version
LiquidCrystal lcd(8,9,4,5,6,7);

// MAC address to use for Ethernet adapter
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };

unsigned int localPort = 8888;       // local port to listen for UDP packets

char timeServer[] = "time.nist.gov"; // time.nist.gov NTP server

const int NTP_PACKET_SIZE = 48; // NTP time stamp is in the first 48 bytes of the message

byte packetBuffer[ NTP_PACKET_SIZE]; //buffer to hold incoming and outgoing packets

// A UDP instance to let us send and receive packets over UDP
EthernetUDP Udp;

const char *szMonth[] = { "Jan", "Feb", "Mar", "Apr", "May", "Jun",
                          "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" };

void setup()
{
  // Open serial communications and wait for port to open:
  Serial.begin(115200);
  Serial.println("Setting up ethernet");

  // Init the LCD display
  lcd.begin(16, 2);
  
  // start Ethernet and UDP
  if (Ethernet.begin(mac) == 0)
  {
    Serial.println("Failed to configure Ethernet using DHCP");
    // no point in carrying on, so do nothing forevermore:
    for (;;);
  }
  Udp.begin(localPort);
}

unsigned long secsSince1900 = 0;

int hour = 0;
int minute = 59;
int second = 58;
int day = 0;
int month = 0;
int year = 1900;
int dow = 0;
boolean fDST = false;

char buf[256];

void loop()
{
  // Once every hour, update from the net, this takes a couple seconds so we don't do every time thru loop
  if (minute == 59 && second == 58)
  {
    sendNTPpacket(timeServer); // send an NTP packet to a time server
  
    delay(1000);
  
    if (Udp.parsePacket())
    {
      // We've received a packet, read the data from it
      Udp.read(packetBuffer, NTP_PACKET_SIZE); // read the packet into the buffer
  
      // the timestamp starts at byte 40 of the received packet and is four bytes,
      // or two words, long. First, extract the two words:
  
      unsigned long highWord = word(packetBuffer[40], packetBuffer[41]);
      unsigned long lowWord = word(packetBuffer[42], packetBuffer[43]);
      // combine the four bytes (two words) into a long integer
      // this is NTP time (seconds since Jan 1 1900):
      secsSince1900 = highWord << 16 | lowWord;
      Serial.print("Seconds since Jan 1 1900 = ");
      Serial.println(secsSince1900);
    }
  }

  // Unix time starts on Jan 1 1970. In seconds, that's 2208988800:
  const unsigned long seventyYears = 2208988800UL;
  
  // subtract seventy years:
  unsigned long epoch = secsSince1900 - seventyYears;

  hour = (epoch / 60 / 60) % 24;
  minute = (epoch / 60) % 60;
  second = epoch % 60;

  // Algorithm from http://howardhinnant.github.io/date_algorithms.html#civil_from_days
  // an era is a 400 year period starting 1 Mar 0000.  
  long z = epoch / 86400 + 719468;
  long era = (z >= 0 ? z : z - 146096) / 146097;
  unsigned long doe = static_cast<unsigned long>(z - era * 146097);
  unsigned long yoe = (doe - doe/1460 + doe/36524 - doe/146096) / 365;
  unsigned long doy = doe - (365*yoe + yoe/4 - yoe/100);
  unsigned long mp = (5*doy + 2)/153;
  
  month = mp + (mp < 10 ? 3 : -9);
  day = doy - (153*mp+2)/5 + 1;
  year = static_cast<int>(yoe) + era * 400;
  year += (month <= 2);

  dow = dowFromDate(day, month, year);
  fDST = isDST(day, month, dow);

  // Build the print buffer
  sprintf(buf, "%2d:%02d UTC %2d %s", hour, minute, day, szMonth[month-1]);
  lcd.setCursor(0, 0);
  lcd.print(buf);

  // Print the second time zone
  if (hour < 8) hour += 24;
  sprintf(buf, "%2d:%02d %s  %d", hour - 7, minute, fDST ? "PDT" : "PST", year);
  lcd.setCursor(0, 1);
  lcd.print(buf);
  
  // Keep the network alive every 60 seconds
  if (second % 60 == 0)
      Ethernet.maintain();
      
  // Accuracy depends on system clock accuracy, corrected once an hour.
  delay(1000);
  secsSince1900++;
}

// send an NTP request to the time server at the given address
void sendNTPpacket(char* address)
{
  // set all bytes in the buffer to 0
  memset(packetBuffer, 0, NTP_PACKET_SIZE);
  // Initialize values needed to form NTP request
  // (see URL above for details on the packets)
  packetBuffer[0] = 0b11100011;   // LI, Version, Mode
  packetBuffer[1] = 0;     // Stratum, or type of clock
  packetBuffer[2] = 6;     // Polling Interval
  packetBuffer[3] = 0xEC;  // Peer Clock Precision
  // 8 bytes of zero for Root Delay & Root Dispersion
  packetBuffer[12]  = 49;
  packetBuffer[13]  = 0x4E;
  packetBuffer[14]  = 49;
  packetBuffer[15]  = 52;

  // all NTP fields have been given values, now
  // you can send a packet requesting a timestamp:
  Udp.beginPacket(address, 123); //NTP requests are to port 123
  Udp.write(packetBuffer, NTP_PACKET_SIZE);
  Udp.endPacket();
}

// In most of the US, DST starts on the second Sunday of March and ends on the first 
// Sunday of November at 2 am both times.
bool isDST(int day, int month, int dow)
{
  if (month < 3 || month > 11) return false;
  if (month > 3 && month < 11) return true;

  int prevSunday = day - dow;

  // In March, we are DST if our previous sunday was on or after the 8th.
  if (month == 3) return prevSunday >= 8;

  // In November, we must be before the first Sunday to be DST which means the
  // previous sunday must be before the 1st.
  return prevSunday <=0;
}

// Day of week calculation
int dowFromDate(int d, int m, int y)
{
  return (d += m < 3 ? y-- : y - 2, 23*m/9 + d + 4 + y/4- y/100 + y/400)%7;

}

Tuesday, April 18, 2017

Arduino IDE issues

Well, it has been a long time since I blogged anything so I thought I would jot down some notes about my experiences of late using recent builds of the Arduino IDE.

I have over the last couple of years been working almost exclusively on Linux workstations, specifically Ubuntu 16.04 LTS and later and my comments are based on my experience on that platform.  I have not yet attempted to reproduce any of this on Windows though I have no reason to doubt that experience would be the same.

In my case, I have loaded up Arduino 1.8.1 on Linux and I have several comments about the later builds and this one in particular.

  1. There has been a lot of work done on board management and library management and this from my perspective has been a very worthwhile addition to the system.  Having the ability to be notified when updates have been produced to the set of development boards supported as well as the libraries is a huge improvement over the older experience.  Trying to make sure you get a library zip file loaded in exactly the correct folder without destroying the base install is a welcome addition.
  2. In the same vein, there has been a bit of expansion on the scope of the 8 and 32 bit processors supported.  This support is bundled up in packages that can be individually installed to add or remove support.
  3. Someone finally addressed the issue of the serial monitor and serial plotter output windows having to be closed in order to recompile and upload the code.  Thank you!
There has also been a lot of work in turning on (by default) the various code optimization flags for the compilers and this has resulted in a lot of code space savings for my projects and I know for others.  It has not come without a price however and this is one of the things I want to highlight here.

One thing I have noticed is that the "link-time optimization" flags have been added to the default options during compile and link.  If you enable verbose output in file->preferences dialogue shown below, you can see these additional command line options during compilation.



While I am a fan of compact code, enabling these options on one of my projects has resulted in the code linker seg faulting during compilation.  I have not yet created a simple repro of the issue.  The following code illustrates the issue, but does not fail to compile, so my understanding of the issue is as yet incomplete.  I wrote this to try and reproduce the issue, but it fails to fail...


class data
{
public:
    int a;
    int b;
};

class classB
{
public:
    void doSomething(data *pData);        
};

class classA
{
public:
    void doit();
    
private:
    classB b;
    data d;   
};

void classB::doSomething(data *pData)
{
    pData->a = 1;
    pData->b = 2;
}

void classA::doit()
{
    b.doSomething(&d);
}


classA *pA;

void setup()
{
  pA = new classA();
}

void loop() 
{
    pA->doit();

}

The basic idea is that I have a class (classA) that has two other classes embedded in it (classB and data).  ClassB has a method (doSomething) that gets passed a pointer to class data which it attempts to use to access public variables in class data.  In my failure case, if I comment out the call to b.doSomething(&d) everything compiles fine.  If I uncomment it, the GNU linker Segment Faults and terminates.  The code above does not reproduce the symptoms.  In my project that does reproduce the problem, if I remove the -flto command line option to disable the link-time optimizer, there is no crash and with the option set it does crash.

There is a lot of discussion on the GNU tools blogs about this problem though my repo seems to be somewhat unique, so I have not been able to narrow the problem down further.  https://github.com/arduino/Arduino/issues/660

My temporary solution is to disable the link time optimizer and finish up the project.  Once it is complete, I will re-enable the option and see if the problem still reproduces.  I am not quite ready to publish this project, so I am going to wait until it is finished before reporting a bug back to GNU as I will have to publish the code to them to get it investigated.  Otherwise, I will build the tool myself from the source and debug it myself.  I just don't want to take the time at this moment with a side trip into debugging my toolchain.

Mostly, I wanted to publish this so that readers will be aware of it and if anyone has encountered the same problem, I would love to compare notes.