Add Development tools to the PiServr

As a developer, I would like continuous integration set up on my home server. I am most familiar with Jenkins, and that is part of the debian distro – so I decided to go that route.

$ sudo apt-get install jenkins

Connect via localhost, and configure security (before the next part). use local database, and matrix based permissions. Make sure to “allow users to sign up”, and add read overall permission to anonymous before adding another user and giving them full permissions. Then add yourself (as the full permissions user). Then you can go back and disable allowing users to sign up.

Edited /etc/default/jenkins to enable listening on the actual network interface so you can connect from your network (and port-forward your router to connect from the internet).

Posted in technical | Leave a comment

Privacy, security, low cost – The Self Hosted PiServr.

Your very own slice of Raspberry Pi can host all your personal cloud needs, with real privacy, for less than $100 all told.


Pictured above is a Raspberry Pi single board computer sitting atop a Western Digital Pi Drive – in my case a 1TB disc, which hosts this blog and a few other services.

A complete server solution, including MariaDB (open source SQL) and Apache web server. Not on the cloud. In your living room, or your office shelf. Consuming a few measly watts – cents per month. No Amazon bills, none of your data need be in the hands of the data giants. Now you can have your Pi and eat it.

Hardware costs in USD:

Raspberry Pi                          35.00
WD PiDrive Foundation edition 250 GB  28.99
Blue Square enclosure                 14.99
3A Power supply                       12.99
Total                                 91.97

The 1TB drive I use is available for a mere $30 more.

What can it do? I have more than 600 GB of files (mostly movies and music) on it. It hosts my blog sites, and a mediawiki site which I use for project documentation. I have two printers (an HP laser and an Epson inkjet) connected to it, so we can print from anywhere in the house (including from my phone or Chromebook). I can play my music through a connected stereo. I can even access the Pi remotely, using either ssh and a command line or using X11 forwarding to run graphical software from any X Window capable computer (running Linux or Cygwin X-Window).

This project will make available a downloadable image, which you copy to a micro-SD card. This image boots your Pi. Then you connect to your Pi using a web browser (Chrome, Firefox) and enter some simple configuration details to enable each web service. Modify your router’s forwarding to connect your Pi’s services to the internet. Then use a dynamic DNS service to locate your server from the outside world.

Posted in technical | Leave a comment

Now super secure-ish!

Whoopie! I have just successfully installed strong ssl encryption (https) for this WordPress blog. Thanks entirely to the great folks at letsencrypt – with a generous G+ prompt in my feed from +Steven Vaughan Nichols. I say Secure-ish because I hate to tempt fate (or hackers) with over confidence, and as has been said, WordPress has a reputation for insecurity.

Posted in technical | Leave a comment

Ram 1500 ecodiesel review – 6 months, 30,000 km

I have owned Big Blue, my 2016 Outdoorsman ecodiesel for just over 6 months, and driven just over 30,000 km. I have made one significant trip – driving from Hamilton to Orlando and back for a disney vacation, made my daily 65 km commute, and towed Marshmallow (an 11 year old Percheron Mare) 3 or 4 times in her gooseneck trailer. My daughter is a frequent rider in the back of the Quad cab, as well as a couple who sometimes ride-share with me on the work-commute.

Throw in sundry trips to the hardware store (4 x 8 ply, 12′ cedar planks etc), grocery shopping, taking the dogs to the park and you get the life of Big Blue.

She is the Outdoorsman rig, with 4×4, skid plates, tow hooks, trailer mirrors, trailer tow brakes, limited slip diff, Quad cab (like extended in other brands), 6’4″ bed, in dark blue (Blue Pearl) with black lower panels and grill. I took the Comfort package (I hate the cold wheel in the winter), and the deluxe package (which had the multimedia options I consider necessary these days), and the electric sliding rear window. The only option I am not convinced of is the rear window. I did not take the backup camera (and it might have saved me a replacement taillight – cost 175). One serious gripe I have is with the ordering process. Both the online customer facing build tool, and the salesman’s ordering tool, showed that the outdoorsman package would come with the low ratio rear axle (for greater towing capacity). However, the higher ratio was delivered, and we were never able to resolve why, or make some satisfactory arrangement to compensate for what I felt was a mistake on their part.

I have not babied the truck – I haven’t really washed or waxed her since owning her, and I have made small “bondo” patches to the rear after scraping it a couple of times (and busting a taillight) due to my underestimating her size.

My average MPG is 25.56 Miles per US gallon (30.7 miles UK gallon), or 9.2 l/100 km overall. This includes all the driving types I mentioned, over a total of 30,302 km. My mileage has been tracked on, a handy web site that enables you to track mileage fill-up by fill up.

So far, I have had 2 oil changes – every 12000 km. These have been quite expensive – the first at $175, second was around 350 as they did some other maintenance work – tire rotation, brake, exhaust and suspension inspection. I am left not terribly happy about the second service – I will double check the invoice for the work done. That said, the truck has performed perfectly, with 0 problems. I had a gooseneck hitch fitted (by Hitch City) a couple months ago for trailering our horse, which cost around $1000. The next thing I really need to do is to get the truck lined. I plan to buy a bucket of the liner stuff and paint it myself (being a cheap Yorkshireman as my wife says). I have found a source for oil and filters online which would cost me about 150 per oil change – not much saving, but I sometimes like to do it myself anyway. Before our disney trip, I purchased a soft tonneau cover online for about 275.

So now for the driving. The truck rides really well – smooth and well behaved, not jittery and “truck like”, thanks to the all round coil springs. Under load, or towing she rides even better – towing the gooseneck with Marshmallow aboard with great aplomb – all in a days work, and averaging under 15 l/100 km while towing her. The bed lights are great for fishing stuff in and out of the back at night. There is sufficient room in the back even for occasional adults, and I am happy that I chose the Quad rather than the Crew cab – the bed length is good. If the truck was any longer, parking would be next to impossible on my street. As it is, I have to park within an inch to avoid sticking out fore or aft over my neighbour’s driveways.

Acceleration is somewhat modest from start (I was driving a Fiat 500 before this truck), but passing at speed seems fine, and towing power feels good. Turbo lag is somewhat pronounced (compare to my wife’s VW diesel), but not a problem when you get used to it. The brakes are great. The tires are pretty grippy in poor conditions – she felt really sure footed in the tail of of Winter in March earlier this year. The driving position is great, with a commanding view of the road, and reasonable over the shoulder visibility. The mirrors are really good, almost eliminating the blind spot. The seats – firm fabric – are comfortable and very adjustable including lumbar support – good for my back considering how much time I spend in this truck – about 3 hours daily.

The Disney trip was a good test at 2 months and about 7,000 km in. I got my best fuel tank on that trip, making 8.3 l/100km (28.3 mpg US or 34 mpg UK) on a stretch coming back up the coast through South Carolina and Virginia. She pulled up and down the Apalachians like a champ, the 8-speed gearbox switching away handily, but never making me feel unnecessarily. The ride was comfortable (as a 10 hour a day drive can be). On my daily commute, I still average about 9 l/100 km, on an average speed of about 80 km/h (mostly 100 but some start-stop in traffic).

As a cost-benefit analysis of the diesel, I look at for my consumption. I have paid $2,359 in fuel over 30,302 km. If I had purchased the 5.7 Hemi (the pentastar would not have met my towing needs), with an average consumption of 14.7 l/100km, I would have spent about $4,000. So I saved about $1600 in 6 months. Factor in the extra cost of the diesel – $5,700 – 1400 for the hemi = 3300. It will take me less than 3 years to pay off the initial investment. However, since I financed the truck anyway, my monthly fuel savings are way more than the extra monthly billing for the diesel option. In other words, I actually have about $200 extra in my pocket every month when adding both the monthly payments and the fuel costs together. Mind you – I don’t have a hemi – but I consider that a small price to pay next to the savings I am enjoying.

Posted in technical | Leave a comment

Upgrading Mediawiki

So very recently mediawiki became “no longer supported” in debian Jessie. This coincides with the fact that the version in Jessie is a venerable 1.19 (current version is 1.26). I discovered this while trying to get a flexible banner extension working (WikidataPageBanner), and it wouldn’t play nice. So I decided to upgrade my wiki.


Here is the Wiki about upgrading the Wiki! (Proof that it worked).

The one thing I wish I had done before starting is to prevent my ssh connection from timing out. It ended up backgrounding the database backup task and doing stuff every minute just to make sure the shell stayed open.

Posted in technical | Leave a comment

Choosing a php Framework

I have an old (10 years) web application that was designed to build, track and print characters and create spell lists for the 3rd edition of Dungeons and Dragons.

The application is written in php, without a framework, with a MySQL backend. It still works, and prints out nice pdf character sheets, but I don’t play 3.x anymore. We have (happily) transitioned into 5th edition, and I miss the functionality. So my main task consists of a re-write for 5th edition. At the same time, I want to use a framework, both in order to learn something new (I know, sucker for punishment), and to make it more maintainable going forward.

I did a bit of comparison “shopping”, and ended up taking a short dive into zend. Unfortunately, I found zend to be a little too nuts and bolts. Although I was coming from hand coding the whole thing, I have played with Rails, so I was expecting a little more framework than zend seemed to offer (and frankly, better or easier to follow docs). I am now trying Symfony, and I am much more impressed. It seems to have a very rails-like philosophy and layout, and I actually prefer php syntax to ruby. To me, ruby seems like a programming language written by someone who has a grudge against C, whereas php is unabashedly rooted in C like syntax, which is familiar to me.

I just noticed one weird side effect – all my (self hosted) sites now show up with the Symfony logo on the tab. Must be an apache configuration screw up!

Watch this space for details of how this progresses.

Posted in technical | Leave a comment

The TissueCam

An idea just came to me how I could package the dashcam case – in a tissue box! I could make one of those open-bottom tissue box covers that can sit on the dash (many cars come with a recess to hold just such a box).

That way, the Pi can be subtly hidden in plain sight. The box would be slightly (the thickness of the Pi board) wider towards the windshield, It would hold a normal box of tissues. The streaming interface would be used to view the camera in real time to adjust its position. The main challenges would be connecting to power inconspicuously – although it could be a handy USB hub, and keeping it stable and not sliding around. Some inconspicuous rubber feet, and some solid (laser cut plywood) construction should do the trick, I think, as well as look pretty good.

The size of a low profile tissue box look like 4 3/4″ x 9 1/2″x 2 7/8″ tall inside dimensions, which should hold a Raspberry Pi nicely when mounted vertically. The outside dimensions of my box would be about 6 1/4″ x 10″ x 3″ when constructed with 1/4″ ply.

View from below (not to scale):

+----( )-----------+
|    cam   ---Pi++ |
|                  |
|    (tissues)     |
|                  |

I would have enough room for the power supply board and supplementary battery. The Lens would fit into a hole on the front edge (more to the centre than my picture above). I will post a better diagram once I have sketched out the plan to scale.

Posted in technical | Leave a comment

Pi Dashcam Software

Having pieced together the hardware for my development system (Raspberry Pi Model A, Pi Camera, Microstack GPS module, SunAir power supply, Edisoft WiFi), it is now time for me to create the software.

Here is my feature wish list, in roughly priority order.

  • Forward facing camera
  • Save video recording to USB (or server via WiFi hotspot)
  • Save up to preset maximum video, in a ring buffer unless record button pressed, which will save previous clip plus next 2 clips permanently
  • GPS and timestamp to video
  • Rear facing camera (USB)
  • Recorded video playback on network device.
  • Live stream capture to a cell phone display.
  • Automatic data transfer to server via network.
  • Local battery, smart charging.
  • Accelerometer activated record mode.
  • Unattended car mode.

Forward facing camera.

The forward facing camera is the official Pi Camera, so the python picamera module will do all the heavy lifting (did I mention that I intend to write everything in python?

On startup, the unit should check for USB storage available (and if so, create directory tree if non-existent), and do all recording to USB. Otherwise, recording should happen to local (SD) recording directory.

Implementing a simple ring buffer should be straightforward. I will capture video in 20 second chunks continuously. When the ring buffer is nearly full, I will delete the oldest chunk and continue. If a record event occurs, I will link the previous and the current chunk to the save folder, and setup the next chunk to be linked to the save folder, until the stop recording button is pressed. It occurs to me that the start and stop buttons can be the same button, and that a double press will work to save the previous current and next (3 segments of video). An LED to show that we are saving video would be a nice addition to the hardware. I could use the camera LED to indicate this, as long as I make the LED visible to the driver.

GPS and Timestamp.

Using the GPSD daemon and it’s python module, I can get the current Lat/Long and I can use the GPS time to set my system clock (since the Pi lacks a real time clock).

The picamera module include the ability to add a line of text overlay using the annotation feature. We can use this to provide a timestamp and GPS position information (and speed) to the recording.

Rear facing camera.

To use a USB camera (cheap webcam) from python, OpenCV has a suitable python module. It is (perhaps) a pity that we need two different APIs one for each camera, but such are the vicissitudes of life.

Recorded video playback on network device.

Use recording to network stream, or file mounted by NFS?

Live stream capture to a cell phone display.

Hmm. Tricky? It seems we can create a custom output stream which wants a write and a flush method. We can use this to write to a file, and to a network stream with the same frame. I will need to play with this to see how it works (and how well it works).

Automatic data transfer to server via network.

On connection to local WiFi, a dashcam script can copy all recorded video chunks to a predefined server location (may use ownCloud for this).

Local battery, smart charging.

Using the SunAir card, I can connect a mini-usb for normal power, and a LiPO cellphone battery for backup. The SunAir will charge the battery when the car is running. The SunAir will power the Pi from either the USB or the battery (when the ignition is off). I hope to find a signal from the SunAir card that will tell me when USB power is interrupted so I can perform a smooth shutdown of the Pi.

Accelerometer activated record mode.

Automatic recording when a significant accelerometer event occurs (strong braking, sudden change in direction etc). There is a python module for using the accelerometer.

Unattended car mode.

Mode to record video when a vibration occurs. Perhaps this will be normal operation with file save triggered by a very low threshold accelerometer event, or even a sound event (captured by webcam microphone).

Posted in technical | Leave a comment

Minimal Pi Dashcam

Here I am creating a dashcam with simple software features and a minimal UI. One may think that this is pointless given the plethora of cheap devices already available. Indeed, the simple designs I have seen can be had for as little as $30. However, a two camera design competes with dashcams that cost over $200, and with added GPS, the home build becomes competitive. The new release of the Pi zero now has added camera support! I will continue prototyping with the Model A, and switch to the zero when it arrives (albeit I had to subscribe to the magazine to get it).

Main component pricelist:

Raspberry Pi zero      5    Raspberry Pi (via magazine)
Pi Camera             25    Raspberry Pi (on hand)
GPS module            25    Microstack (on hand)
WiFi adapter          10    Edisoft (on hand)
USB camera            10    (Tiger Direct)
Wide angle lens       10    Aukey kit (Amazon)
Power supply          25    SunAir (on hand)
Battery               free  Old cell phone (on hand)

Obviously these are retail module prices, and a finished manufacturing design would have much cheaper components – but that is unlikely to happen, since this is just a hobby and I lack the resources to compete with manufacturing companies.

At this point, it may or may not have a display (I have a handy 2.8″ 320×200 display from Adafruit). As a headless design, it could connect using WiFi, and push chosen video to a cloud drive (such as google drive, or ownCloud), or simply save to a USB key. It could use a model A (with WiFi or a USB stick), or the latest model 3 B (which has WiFi included), or even the elusive Pi Zero. A benefit of a headless design is that the only component that would have to be visible would be the camera (and it’s cable).

Conceptually, it will take video in x second chunks, in a loop of y segments. On the “save” button press, it will save the last y segments, and continue saving for another z segments – where x, y and z would be programmable parameters. A rear facing camera would be a nice addition (using USB since the Pi only supports a single CSI camera interface). Audio would be another option (requiring a microphone – but the rear USB cam might have one included). Conveniently, I think my truck would make mounting a rear camera fairly easy.

The user can view video using their existing mobile device or computer screen, either from the USB stick, over WiFi from the cam, or their cloud of choice.

I intend to use this guide as the starting point. I would gladly credit the author, but I can’t establish his name.

So far, I have installed minibian on an 8 GB SD card. I have expanded the filesystem, renamed the system to “dashcam”, enabled the camera and also SPI and I2C (required for the Adafruit screen), and updated the software. I made the necessary modifications (from Adafruit’s website) to the kernel and boot config to drive the display. I also installed fbcp – a tool which copies the internal GPU frame buffer to the SPI console frame buffer – required for the SPI driven screen to show the Pi Camera output in real time. This may go away if I switch to headless design. Initial tests show the video from the camera on the display. I installed omxplayer to allow local replay of the h264 video produced by the raspivid tool.

In terms of hardware I have ordered a set of camera lenses from Aukey to get a wider angle on the Pi Camera. I could have gone cheaper, but I wanted glass lenses. I have a small camera bracket, and a couple of simple Pi cases I can use until I have settled on a configuration and mounting method.

I want at least two buttons – one to trigger saving video, and one to initiate shut down. If I go with the touchscreen, it has optional physical buttons already. Otherwise I will use buttons on GPIO lines and project them through the case.

I have a GPS module which I can integrate, software can overlay the time and coordinates on the video. If I take this route, I am strongly considering adding navigation software and a larger screen (7″ Pi Display) to make full use of it. NAVIT sounds like a strong contender for this. However, I shall continue building from small and see how it works out. Another piece of hardware to consider is an INU. The accelerometer can be used to trigger autosaving – in the event of a crash for example – although I suspect a simple tilt sensor would also work for this purpose. As well, an INU can be used to augment or smooth GPS data using a Kalman filter – although I am not sure whether (or how hard it would be) to integrated that into NAVIT.

I successfully hooked up my gps module (a microstack board), by connecting the 3.3, ground and TX/RX lines to my Pi headers. One issue I ran into with gps is the gpsd package does not much like systemd. On initial install, it works fine, but as soon as I reboot, it does not start. running dpkg-reconfigure gpsd starts it up again, but again it fails on reboot. I followed the instructions here, and replaced systemd with tradition sysvinit support, and my problems miraculously vanished.

I then discovered another problem – raspbian jessie comes with libgps21 and the dashcam scripts were built for version 2.2. Anyhow – I think I need to go beyond the scripts from the pidashcam guide. I think I will write a python application from the ground up, using the picamera module, gpsd.

Last but not least, I need to look at the power supply. Car cigarette lighter chargers usually drop abruptly as soon as they engine is stopped – in order to protect the car battery. This could be disruptive to data collection, and possibly injurious to the SD card. Ideally, I would have a constant 5 v supply available, and a notification when ignition goes off, in order to shutdown gracefully. I am considering tapping into the overhead cabin light supply (as this is available after ignition off), and find another line in the vicinity to tell me when ignition is off.

On further consideration, I realize I have a SunAir Solar Power Board kicking around – I bought it on a kickstarter some time ago. It is a $25 board that controls charge to a 3.7 V LiOn battery, and supplies 5 V to Raspberry Pi-like devices. It expects 6 V from a solar panel, or a mini usb connector to charge the battery / power the device. This, coupled with a cell-phone battery or two will ensure I have a smooth shutdown, or ignition-off power available. The downside of this device is that it supplies only 1000 mA, which would only supply the model A + camera safely.

Posted in technical | Leave a comment

A Delicious Serving of Raspberry Pi

You are viewing a WordPress blog post served up by a Raspberry Pi 2 Model B sitting in my living room.

This post is living proof that I have completed an initial foray into some serious Raspberry Pi server functionality, albeit with a few glitches (photos are not now showing). I have not yet gotten a case for the Pi Drive – it is resting in the foam packing it shipped with. I am looking at either printing one – or using the small stacking case from WD themselves.


I was only intending to test operating systems, but my primary file and print server (of the last 5 years) went down with a power outage, and didn’t come back up properly. I could boot it with a CDROM, but while attempting to fix its boot problems (and it is using RAID5 with 3 hard drives), I munged up the OS and got into apt-get hell. So I thought to myself – why not expand the scope of the project? It’s not like I don’t have enough on the go, right?

I had a lovely little piece of kit from Western Digital – a Pi Drive, which is a 1 TB hard drive in 2.5″ form factor, with a nice little Y cable and a power supply for both the Pi and the drive. Turns out I only have 500 GB of data, so this is perfect for now. If all else fails, I can use the drive as a backup and I can wipe the big server and reinstall.

Note that I have a Wi-Pi Wifi adapter (which I later remove as I am not using it) in the picture. Strangely, the much smaller Edimax adapter gets a better signal than the Wi-Pi. I will also be ditching the display, and the large(ish) box, as I don’t need it – and both are designated for another project.

Pi model 2 with Pi Drive and 7 in Display

However, I am a real cheapskate, and an environmentalist (if for no other reason than I am a cheapskate). It has always bugged me that my server chews down over 100W continuously, 24/7, 365 days a year. The Pi draws about 1W when idle, and the Pi Drive probably about the same. So I thought to myself – lets see if we can get file and print serving up and going, see what performance is like, and then I can futz around porting web-site stuff and anything else I used to rely on the old server for. I didn’t want to get bogged down learning a new distro’s package manager, but I had noticed a new kid on the block in the form of Minibian – a stripped down version of Raspbian, my favourite Pi OS. I gave that a shot – since it uses the familiar debian package manager. It has not let me down. There were no issues whatsoever in getting packages installed and configured to my liking.

My first priority was getting my data copied over. It took about 11 hours to copy over 500 GB of data (mostly videos and music). I quickly set up NFS exports, so now I can get to my stuff from all the other computers as before – except if I boot Windows (which is rarely). I still need to setup Samba to complete this task. While still on the basics, I configured cups for printing from my Linux machines – and to my joy I also discovered a ChromeOS extension for printing to cups. That – after some configuration glitches – was accomplished, so now I can print from Linux or our Chromebooks, 90% of our needs. Samba will deal with another 9%. The missing 1% I hear you ask? I will need to install the Chrome Browser and add the Google Cloud Print extension. Hey – I just discovered the Google print cups connector!

I have also installed owncloud – a personal cloud solution, to wean myself off of reliance on Google and Dropbox. So far, owncloud works pretty well. It even has a document creator – using Libre Office as a back end, although for now it only supports docs, not spreadsheets or slides. I am quite impressed – and it is under an Apache open source license.

Initial trials with my web sites were a little more discouraging. I got blank pages, or only the static content. I decided while doing all this I might as well switch to mariadb instead of mysql. That installed fine, and I simply manually copied my ISAM and InnoDB tables over. I then basically ported all my apache “sites-enabled” over, put the right passwords in the right places, and very little worked. I fiddled around for a day or two – before I discovered that I had a broken wordpress theme. I activated the twenty-fifteen theme and suddenly all my sites worked. So – wordpress is working well – this post is on the new WordPress Blog, successfully ported to a Raspberry Pi 2. Soon after that, I installed the current version of the twenty-ten theme for which all my header images were customized, and we were gold.

Now for my old php driven D&D sites. I maintain a 3.x edition character database and pdf producer of nice character sheets and a spell database which is great for choosing and printing spell books. I want to upgrade these to do both Pathfinder and 5th edition. So far I have partial success. I am confident I will soon have everything working.

I would also like to get an e-mail server working. I used to run my own for about 10 years, on a Linux server. I gave it up when google became the defacto provider, and I had moved once too often to want to be without e-mail for 3 days while ISPs failed to live up to their installation agreements. I have installed postfix, which came with an unexpected challenge. The upstream debian posfix package had a requirement for either mysql or postgresql. I was using mariadb. After forcing it to install anyway, I still got error messages blocking my progress with every subsequent apt operation. I bit the bullet and hand-edited the .deb package to allow mariadb-server as an alternative. Something the package maintainers themselves ought to do. However, it was quite easy, and I learned something new in the process. All good. I then needed to use the same trick for mediawiki, which installed fine once I added mariadb to the “recommended” list in the control file.

I will add posts giving detailed technical HOWTOs for various portions of this project if people show any interest. Let me know!

Also based on interest – I will create a unified configuration system for all the local configuration (server name, domain, user logins, outgoing smtp etc), and an SD image for download. I may need to create a kickstarter for this (at least so I can justify all the time to my wife).

Posted in technical | 3 Comments