Using Fail2Ban to protect your LAMP application

From the Fail2Ban Homepage:

Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the  malicious signs — too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, courier, ssh, etc).

The Problem

When a web site comes under attack, occasionally, the resources of the machine become exhausted creating a denial of service.  This, possibly, is not the aim of the attacker but a side effect of their Good Work.  Without an investigation that would find the attacks the application’s developer might ask for the hosting machine to be given more resources.  More of a resource, like RAM, would help the application to avoid the denial of service but would afford the attack more time in which to be successful.  And, of course, this would be expensive.  The increase in RAM could be used elsewhere and the machine would not always need the extra resource because the attacker will move on if the application is esoteric or reasonably well written.  It is easy to give extra RAM to a machine but it is harder to take it away.

These attacks are attempts to penetrate a system through a flaw in the application or operating system.  Most of the attacks we see are against popular applications: phpMyAdmin, WordPress, and their Windows equivalents.  As long as good practices, when developing a LAMP application, are used e.g. using prepared SQL statements to prevent SQL injection, the main application might be safer from attack than are the more popular well understood applications.  This situation is weakened when a utility like phpMyAdmin is used to provide functionality not yet present in the main application.

So, instead of adding more RAM to mitigate against an accidental denial of service and to protect well known applications as well as the main application, we could instead monitor for attacks and proactively protect the host machine accordingly.  This is where we might use Fail2Ban.

The Attack

As most attacks are against the application and most Internet connected application are web applications we are probably talking about Apache or NGINX for delivery of content.  We will spot attacks through the logs files, for example, error_log:

[Sun Nov 05 09:34:27 2017] [error] [client 51.254.130.82] File does not exist: /home/apache/public/htdocs/admin
[Sun Nov 05 09:34:27 2017] [error] [client 51.254.130.82] File does not exist: /home/apache/public/htdocs/admin
[Sun Nov 05 09:34:27 2017] [error] [client 51.254.130.82] File does not exist: /home/apache/public/htdocs/admin
[Sun Nov 05 09:34:27 2017] [error] [client 51.254.130.82] File does not exist: /home/apache/public/htdocs/apache-default
[Sun Nov 05 09:34:27 2017] [error] [client 51.254.130.82] File does not exist: /home/apache/public/htdocs/cpanelphpmyadmin
[Sun Nov 05 09:34:27 2017] [error] [client 51.254.130.82] File does not exist: /home/apache/public/htdocs/cpphpmyadmin
[Sun Nov 05 09:34:28 2017] [error] [client 51.254.130.82] File does not exist: /home/apache/public/htdocs/dbadmin
[Sun Nov 05 09:34:28 2017] [error] [client 51.254.130.82] File does not exist: /home/apache/public/htdocs/db

This shows one IP address (bad machine on the Internet) running through obvious URLs trying to find phpMyAdmin but failing (404).  It shows that there are many attempts in a short amount of time.  Fail2Ban could be set up to ban the bad machine for a period of time after the first three failures preventing the rest of the attempts draining the resource of the machine.  This log does not show actions that might exhaust a machine’s resources but if phpMyAdmin was found a series of tests could then be run against it and that might create a denial of service.

A Solution for this Attack

Fail2Ban is available to CentOS:

$ yum install fail2ban

The interesting aspects of this package for us are:

/etc/fail2ban/jail.d/
/etc/fail2ban/filter.d

I’ll create this in /etc/fail2ban/jail.d/apache-fourofour.conf

[apache-fourofour]
enabled = true
port = http,https
filter = apache-fourofour
logpath = /home/apache/p*/logs/error_log
maxretry = 20
findtime = 120
bantime = 16400

The above isn’t the complete truth…because that would be telling!

And in /etc/fail2ban/filter.d/apache-fourofour.conf

[Definition]
failregex = [[]client <HOST>[]] File does not exist: *
ignoreregex = .*(robots.txt|favicon.ico|jpg|png)

The inspiration for this filter can be found here, created by Dominic Derdau.

So, a quick check to see if we get any hits (against a file we know has some):

fail2ban-regex home/apache/public/logs/error_log-20171112 /etc/fail2ban/filter.d/apache-fourofour.conf

reveals 184 hits.

Let’s enable the fail2ban service and start it.  This machine is CentOS 6:

# chkconfig fail2ban on
# service fail2ban start

and check that the firewall has Fail2Ban gaols set up:

# iptables -L -v -n
.
.
.
 0 0 f2b-apache-fourofour tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443
.
.
.
Chain f2b-apache-fourofour (1 references)

Now, how to test?  A quick script that quits once it is blocked at the firewall:

n=0
while :
do
  echo "Attempt $n"
  let n=$n+1
  curl http://xxx.xxx.xxx.xxx/bob
  if [ $? -eq 7 ]
  then
    echo 'Firewall block!'
    break
  fi
done

The script above may surprise you.  It did surprise me.  I got many more attempts in before I was blocked.  This is because Apache, in this case, will serve pages as fast as it can but will write to the logs like it’s a hot lazy Sunday afternoon.  You can tune /etc/fail2ban/jail.d/apache-fourofour.conf to suit your needs.

Eeek, I’m in gaol:

Chain f2b-apache-fourofour (1 references)
 pkts bytes target prot opt in out source destination 
 17 1020 REJECT all -- * * 10.3.232.4 0.0.0.0/0 reject-with icmp-port-unreachable 
 190 16313 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0

So, this works.  Check that it works, too, after a reboot.  Fail2Ban should start running after iptables otherwise you’ll miss out on the gaol creation.

Well done you.

Posted in ITMS, Linux SysAdmin, Secuity | Tagged , , , , , | Leave a comment

Developing CampusM using Docker Containers

Back in December, I created a mobile app for Android and Apple using Ionic with AngularJS.  I did this using my desktop running GNU/Linux.  Traditionally, when a developer works with lots of technologies through her desktop new work can eventually step on the toes of other projects or impede the new work.  When I work with Windows I have to create a VM with a full windows install plus all of the tools.  To work with Ionic I created all of the tools locally and a VM with 100GB of disk space.  Therefore, if I want a simple life I will create a vm/container per project which means lots of VMs can quickly chew through disk space.  I have just created a Docker container (not-a-vm) so that I can start learning about and developing for CampusM.  I am am the point of writing the first line of code.  The Docker container takes up a whopping 1.1GB.

The following list is pretty close to what anyone else will need to do to copy this work.

On my desktop now, I just need a web browser and editor (Vi FTW).

A list:

  • desktop:
    • docker pull fedora
    • docker run -i -t -d fedora
    • docker ps
    • docker exec -i -t 25efc7b4714d bash
  • in container:
    • as root: dnf install npm
    • as root: npm install -g https://npm.campusm.net/get/aek-cli
    • as root: useradd -d /home/node -c Node node
  • desktop:
    • docker commit 6a6cf3686621 fedora/campusm
    • docker stop 6a6cf3686621
    • docker run -v /usr1/home/williams/Dev/CampusM/Projects:/home/node/Projects -p 5000:5000 -i -t -d fedora/campusm
    • docker ps
    • docker exec -i -t 0c6cac5cf04fa534b26ecb9aee74740c3e2724f6980a981ceed216b5e3129301 bash
  • container
    • as node
      • cd Projects
      • aek create
      • cd projectname
      • aek start
  • desktop

CampusM working through Docker

  • desktop
    • docker exec -i -t 0c6cac5cf04fa534b26ecb9aee74740c3e2724f6980a981ceed216b5e3129301 bash
    • create RunProject in the shared directory
      • #!/bin/bash
        
        su - node -c "cd /home/node/Projects/$1 && aek start"
  • container
    • as root
      • cp ~node/Projects/RunProject /usr/local/bin/
      • chmod +x /usr/local/bin/RunProject
  • desktop
    • now we don’t have to go in to the container to start the project
    • docker exec -t -i 2f85e2474912 /usr/local/bin/RunProject projectname

Well done you.

Posted in app, Development, Docker, ITMS, Linux SysAdmin, Virtual Machines | Tagged , , , , , , , | Leave a comment

Quick note: Monit-er FFMPEG : run ffmpeg as daemon

On CentOS 6, I just set up a script and monit to rerun ffmpeg if it crashes.  We’re using ffmpeg to push video from an IP camera to a CDN.

Screenshot from 2017-04-10 11-51-54FFMPEG

Install NUX repo in order to get FFMPEG:

$ rpm -Uvh http://li.nux.ro/download/nux/dextop/el6/x86_64/nux-dextop-release-0-2.el6.nux.noarch.rpm

Note: the el6 + x86_64

$ yum install ffmpeg

Monit

Monit will monitor FFMPEG and restart it if FFMPEG crashes.

$ yum install monit

User

Please, don’t run this as root. Create a user “scriptcam”:

$ adduser -d /usr1/home/scriptcam -c 'Script Camera' scriptcam

Create script ScriptCam

In /usr/local/sbin/ScriptCam:

#/usr/bin/bash
pid=`runuser -l scriptcam -c 'echo|/usr/bin/ffmpeg -v 0 -i rtsp://x.x.x.x/channels/1 -r 25 -vcodec copy -acodec copy -f flv rtmp://y.y.y.y/zzzzzzz-live/zzzzzzz01 >/dev/null 2>&1 & echo $!'`
echo $pid > $1

Create Init script for FFMPEG

Install to /etc/init.d/ffmpeg:

#!/bin/sh
#
# ffmpeg Startup script for ffmpeg/ScriptCam
#
# chkconfig: - 85 15
# description: Push ScriptCam video to CDN
# processname: ffmpeg
# pidfile: /var/run/ffmpeg-daemon.pid
#
### BEGIN INIT INFO
#
# Provides : ffmpeg
# Required-Start : $local_fs $remote_fs $network $named
# Required-Stop : $local_fs $remote_fs $network
# Default-Start : 2 3 4 5
# Default-Stop : 0 1 6
# Short-Description : ffmpeg daemon
# Description : ffmpeg daemon
#
### END INIT INFO

# Source function library.
. /etc/rc.d/init.d/functions

prog=ffmpeg
pidfile=${PIDFILE-/var/run/ffmpeg-daemon.pid}
OPTIONS=$pidfile
lockfile=${LOCKFILE-/var/lock/subsys/ffmpeg}
RETVAL=0
STOP_TIMEOUT=${STOP_TIMEOUT-10}

if [ "$(id -u)" != "0" ]; then
 echo "This script must be run as root"
 exit 1
fi

start() {
 echo -n $"Starting $prog: "
 LANG=$HTTPD_LANG daemon --pidfile=${pidfile} /usr/local/sbin/FalconCam $OPTIONS
 RETVAL=$?
 echo
 [ $RETVAL = 0 ] && touch ${lockfile}
 return $RETVAL
}

stop() {
 echo -n $"Stopping $prog: "
 killproc -p ${pidfile} -d ${STOP_TIMEOUT} $prog
 RETVAL=$?
 echo
 [ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
}

case "$1" in
 start)
   start
 exit 0
 ;;
 stop)
   stop
 exit 0
 ;;
 restart)
   stop
  start
 exit 0
 ;;
 **)
 echo "Usage: $0 {start|stop|restart}" 1>&2
 exit 1
 ;;
esac

Monitor with Monit

Use monit to monitor the ffmpeg process. Monit will restart the falcon cam if ffmpeg crashes.

Create /etc/monit.d/ffmpeg:

check process ffmpeg with pidfile /var/run/ffmpeg-daemon.pid
      start program = "/etc/init.d/ffmpeg start" with timeout 60 seconds
      stop program = "/etc/init.d/ffmpeg stop"

At Boot

Configure monit to start at boot:

$ chkconfig monit on

Configure ffmpeg to start at boot:

$ chkconfig –add ffmpeg
$ chkconfig ffmpeg on

Extras

$ service ffmpeg restart # stop | start
$ service monit restart # stop | start

Check the monit web site (the user details are in /etc/monit.conf) :

http://localhost:2812/

Posted in Development, ITMS, Linux SysAdmin, university | Tagged , , , , , | Leave a comment

Ionic 2 : how to build three native mobile apps from one code base

Introduction

I have just created an app using early versions of Ionic 2 (Apache Cordova with AngularJS).  The app has been accepted in to Google and Apple’s app stores.  Obviously, this means I am now Ionic 2 1337 and should write a guide on how to develop using the framework.  There are 750 lines of code in my first app, just to say… there are levels of 1337ness.

One of the reasons for writing up this up is that my colleagues should be able to use this and re-create the steps I took.  To make it interesting for me I am going to create a toy app that includes taking pictures.

Hat tips: of all the research (randomly dipping in to the Internet) I did, Josh Morony has been the greatest help through his blog.  Also, it would have taken even longer, health threateningly longer, to get the Windows tools working without Justin James‘s help.

We need tools

To develop three native apps I’m using three platforms: GNU/Linux, a physical Mac, and a Windows VM.  We could get away with just a Mac but I am way more productive with GNU/Linux.  We have not investigated building the app using a service in the cloud like Ionic’s.  I went through lots of iterations of building the tools that I can’t give detailed  and accurate instructions but hopefully through this I’ll show that it is possible to build Android, iOS and Windows Mobile apps.

Roughly:

  • for GNU/Linux building Android : essentially npm, git, Java SDK, Android SDK and a guide.
  • for Apple: npm, git, Java SDK, Android SDK, Xcode, an Apple Id and a guide.  For an introvert like me, I love the licence (without membership) and Id free world of GNU/Linux.  I can just get on and code.  If you don’t already have an Apple Id then nip over to an Apple store and get something for “free” so that when you create the Id you do not need to register a credit card.  I wrote that bit without swearing.  Set up an Apple Id using this method before you do anything else.  Otherwise, you could, like me, spend too much time trying to get things set up comfortably.  This step, for me, included phone calls!
  • Caveat : Windows hates me.  “Windows” seems apt because the OS seems to be a collection a small opportunities where the thing you have tried repeatedly suddenly works.  I’m sure colleagues who use Windows day in day out will have more luck.  The quickest way into Ionic and Windows might be this guide.  Justin James employs Chocolatey to batch install the tools he needs and because of him you can too.

We need magic

In no particular order these tricks might save your day:

  • I use this to build and deploy to a Windows Mobile set to developer mode and connected by USB to my Windows VM:
    ionic run --device -- --phone --arch=arm
  • Re-install plugins if things go awry:
    $ ionic plugin list
    
    On Linux, build a for loop from the output:
    
    $ for i in cordova-plugin-crosswalk-webview cordova-plugin-device 
    cordova-plugin-network-information cordova-plugin-splashscreen
    cordova-plugin-statusbar cordova-plugin-whitelist
    cordova-sqlite-storage cordova.plugins.diagnostic
    ionic-plugin-keyboard ;
     do   ionic plugin remove $i && ionic plugin add $i; done
    
  • Tidy up before release:
    ionic plugin rm cordova-plugin-console
  • package.json includes version numbers which can be fine tuned to get things working and this will update the current project:
    npm install
  • re-installing Ionic can help (as root):
    npm uninstall -g ionic && npm uninstall -g ionic
  • when you create a new project you get a .gitignore for free.  When you build on a different platform you need to check you have installed the necessary plugins and platform to build your app because these will not get added to your project
  • when we build on a new platform we will have lots missing when we pull from git.  The directory “www” was missing.  Ionic claimed that I was not building an Ionic project
  • also, for a first build on a new platform
    ionic setup sass
  • Icons and splash screens are fun.  For Android and iOS you can put an image 2048×2048 for the splash.png in ./resources and then run the following to build all the images the project needs
    ionic resources --splash && ionic resources --icon
  • Older Android versions have a poor web browser.  Install Crosswalk for consistency across versions:
    ionic plugin add cordova-plugin-crosswalk-webview
  • If you get Sass errors try:
    rm -rf node_modules
    npm install

Hello World

Somewhat riskily I’m upgrading to the official release of version 2 final.  This should be fine but I built my first app with rc3 then rc5.  There will be things to learn with the proper release.

Somewhere neat run:

ionic start IonicHelloWorld tabs --v2 --ts

This will create lots and lots of things, we are mostly interested in:

src
├── app
│   ├── app.component.ts
│   ├── app.html
│   ├── app.module.ts
│   ├── app.scss
│   └── main.ts
├── assets
│   └── icon
│   └── favicon.ico
├── declarations.d.ts
├── index.html
├── manifest.json
├── pages
│   ├── about
│   │   ├── about.html
│   │   ├── about.scss
│   │   └── about.ts
│   ├── contact
│   │   ├── contact.html
│   │   ├── contact.scss
│   │   └── contact.ts
│   ├── home
│   │   ├── home.html
│   │   ├── home.scss
│   │   └── home.ts
│   └── tabs
│   ├── tabs.html
│   └── tabs.ts
├── service-worker.js
└── theme
└── variables.scss

and resources, config.xml, package.json.

At this point we can already run our app in a web browser:

ionic serve

“ionic serve –lab” doesn’t work for me yet.

Let’s add a platform.  I’m on GNU/Linux, because it is my Happy Place, so I will add Android:

ionic platform add android

and now we can run our app in an emulator:

ionic run android

AndroidEmulatorThe are two ways to get the app on to a mobile device:

  • copy the android-debug.apk that was built in the last step and copy it on a device, then tell the device to trust apps found outside of the Playstore (Android app store) and install the app.  The signature for the app is the identification of the app.  New versions built in the same circumstances will replace the old version.  An app built with a different signature will appear as a separate app
  • via USB.  Set up the device so that it is in “developer mode”.  Connect it using a USB cable and check with “adb devices” that the computer can see it:
  • $ adb devices
    List of devices attached
    42eb627e device

    and then run “ionic run android” again.  Excited yet?  Maybe not, if you got a blank screen.  The plugin for loop from above fixed this for me

  • Google Chrome web browser has a trick we can use at this point: chrome://inspect will let us inspect and debug our running app.

Change some code

Looking at the tree structure above we can see that src/pages/home contains the home “tab”.  This will not be very exciting but if I go straight to trying to add camera functionality this blog entry is likely to crash and burn.  In the directory we have: home.html, home.scss, and home.ts.  We can change the text in the .html, the CSS in .scss and the Angular JS in the .ts file.  I spent a week learning version 1 of Ionic and I think version 2 will be better.  Let’s change the home tab… First of all run:

ionic serve

and navigate to http://localhost:8100/.  When we change a file the app will reload in the browser giving us the sense that what we change is what we see.  I use vi (Vim) in most situations because it is nearly always available (hey Windows, get with the programme!)  Edit home.html under src/pages/home/home.html.  Change “Welcome to Ionic!” to “My Little Camera App.” and save the code.  We should see the app reload in the browser and instantly display the change:

$ diff src/pages/home/home.html src/pages/home/home.html.orig 
8c8
< <h2>My Little Camera App.</h2>
---
> <h2>Welcome to Ionic!</h2>

Now, just for fun, let make the title green:

$ diff src/pages/home/home.scss src/pages/home/home.scss.orig
2,4d1
< h2 {
< color: green;
< }

Currently, I’m nervous of style changes.  When we imagine the range of devices and browsers, a style change might not be reflected the way we hope it would in every situation.

Time to get funky

Straight from the horse’s mouth, add the camera plugin:

ionic plugin add cordova-plugin-camera

This is where the fun begins.  Desktop web browsers do not have a native camera available to them.  We need to install our toy app on to a device to see the functionality.  I hope we can borrow a device from work and not compromise our own devices.

Next, let’s change “About” to be “Snap”.  This is fairly complex but essentially we are looking for instances in the file system and code that have “About” or “about” and changing to “Snap” and “snap”.  Change files:

$ find src -name '*about*'
src/pages/about
src/pages/about/about.html
src/pages/about/about.ts
src/pages/about/about.scss

and strings:

$ find src -type f| xargs grep -i about
src/declarations.d.ts: Declaration files are how the Typescript compiler knows about the type information(or shape) of an object.
src/declarations.d.ts: They're what make intellisense work and make Typescript know all about your code.
src/declarations.d.ts: To learn more about using third party libraries in an Ionic app, check out the docs here:
src/pages/tabs/tabs.html: <ion-tab [root]="tab2Root" tabTitle="About" tabIcon="information-circle"></ion-tab>
src/pages/tabs/tabs.ts:import { AboutPage } from '../about/about';
src/pages/tabs/tabs.ts: tab2Root: any = AboutPage;
src/pages/snap/snap.html: About
src/pages/snap/snap.scss:page-about {
src/pages/snap/snap.ts: selector: 'page-about',
src/pages/snap/snap.ts: templateUrl: 'about.html'
src/pages/snap/snap.ts:export class AboutPage {
src/app/app.module.ts:import { AboutPage } from '../pages/about/about';
src/app/app.module.ts: AboutPage,
src/app/app.module.ts: AboutPage,

I made those changes so quickly that the Ionic web server has become confused.  Rebuild the app “ionic build android“, restart the server “ionic serve“, and reload the web page.  Now, the icon “i” for Snap does not make sense.  Browse through the icons and choose a new one.  Make the following change to “tabs.html”:

$ diff src/pages/tabs/tabs.html src/pages/tabs/tabs.html.orig
3c3
< <ion-tab [root]="tab2Root" tabTitle="Snap" tabIcon="camera"></ion-tab>
---
> <ion-tab [root]="tab2Root" tabTitle="Snap" tabIcon="information-circle"></ion-tab>

Next, we need to tell the AngularJS to use the camera library, in “snap.ts”:

$ diff src/pages/snap/snap.ts src/pages/snap/snap.ts.orig
5,6d4
< import {Camera} from 'ionic-native';
<

Adding this will change the warning to the user if they were to install the app to their device.  The warning will say that the app wants access to the camera.

Skipping, because this is so easy, I’ll add a few more bit from the tutorial…

$ diff src/pages/snap/snap.ts src/pages/snap/snap.ts.orig 
5,6d4
< import {Camera} from 'ionic-native';
< 
13,14d10
< public base64Image: string;
< 
17,29d12
< }
< 
< takePicture(){
< Camera.getPicture({
< destinationType: Camera.DestinationType.DATA_URL,
< targetWidth: 1000,
< targetHeight: 1000
< }).then((imageData) => {
< // imageData is a base64 encoded string
< this.base64Image = "data:image/jpeg;base64," + imageData;
< }, (err) => {
< console.log(err);
< });

and:

$ diff src/pages/snap/snap.html src/pages/snap/snap.html.orig
10,19d9
< <ion-card>
< <ion-card-content>
< Hello World, this is my camera app
< 
< <button (click)="takePicture()">Take a Picture</button>
< 
< Latest Picture:
< <img [src]="base64Image" *ngIf="base64Image" />
< </ion-card-content>
< </ion-card>

We have linked, I like this bit, the event ‘click‘ and the function “takePicture()“.  Further down we can see:

*ngIf="base64Image"

The “*ngIf” is another bit of magic.  The image is displayed if “base64Imageis true.  Once we understand this we can do all sorts of things using the power of the evil side-effect.  If a variable is shared across the application a variable set in one tab can effect the magic on another tab.

Right, quick, build the app and deploy it to a device.  Plug the device in again test test “adb devices”.  Then run “ionic run android”.  If I have done a good job in explaining this, we should now have a little camera app running on Android.

Git it

As this is a toy app we won’t add this to TFS but we could do this…

Now we have something running it is probably a good time to add it to our Git repo (TFS).  We can develop on GNU/Linux and pull changes to Windows and Apple rather than go through the pain of developing on either of those platforms.  Ionic works well with git because a little code goes a long way.  A new feature can be added as code and push to git and make sense as a unit.  If we could automate the tools we might even see automatic building of the native apps ready for testing.

Another bite of the Apple

If we are lucky, we will have access to an polite but ancient (think hard drive and too little RAM) Mac Mini.  As above we should have all the tools including Xcode installed.

The slow Mac Mini I am using is headless and I use Remmina to VNC to the Mac Mini.  First, we copy the project on to the Mac because we have not set up git for this “toy” application.  In this case it can be useful to use the trick:

rm -rf node_modules
npm install

A good test is to run “ionic serve”.  Once that is working we can run:

ionic build ios

which is if completes successfully will create

platforms/ios/IonicHelloWorld.xcodeproj/

which can be opened as a project in Xcode.  Attach an iDevice to the Mac and then set up Xcode to connect to it:

Mac Mini running Xcode

Mac Mini running Xcode

The device I am using is called TESTBOX21 which whould be in the drop down circled.  When we select it Xcode will do some jiggery pokery.  Click to edit the project and set “team” to an Apple Id.  Then click “play”.  Shortly, or longly for this Mac, we will be asked to allow the app to run on the iDevice.  On the iDevice go to Settings > General > Device Management.  Click on the Apple Id that you used, trust it and verify the app.  The app is install but we can click on “play” again.  Once again, we have a native version of our app but for iOS this time.

I’ve got that Windows feeling ♪ ♫ ♬

The toy app’s code is not in Git.  I’m stymied.  How do I get the code on to the Windows VM?  Apple has SSH.  The advice is to set up Samba on the GNU/Linux host.  Luckily, we have some Windows infrastructure and I can do two copies to get going…  I might take up knitting.  These copies are going very slowly…bigly slow.  I Ctrl-Ced that and put the code up to DMU’s open git server.

On the Windows VM:

git clone https://git.dmu.ac.uk/appdev/MyLittleCameraApp.git

Tell the project about Windows 10, edit using ‘code‘ config.xml:

$ diff config.xml config.xml.orig 
22d21
< <preference name="windows-target-version" value="10.0" />

Connect a Windows device to the Windows VM via USB.  Go in to the directory and :

npm install
ionic platform add windows
ionic plugin add cordova-plugin-camera

Finally, run the magic:

ionic run --device -- --phone --arch=arm

And there you have it.  One set of code running on three devices.

Posted in app, Development, ITMS | Tagged , , , , , , | Leave a comment

Micro blogging server setup

On Fedora:

  • dnf install docker
  • systemctl start docker
  • docker run hello-world
  • docker pull turnkeylinux/gnusocial-14.0
  • CID=$(docker run -i -t –name social -p 10.12.55.112:4080:80 -p 10.12.55.112:4443:443 -p 10.12.55.112:4020:12320 -p 10.12.55.112:4021:12321 -p 10.12.55.112:4024:12322 -d turnkeylinux/gnusocial-14.0)
  • CIP=$(docker inspect –format='{{.NetworkSettings.IPAddress}}’ $CID)
  • docker logs $CID | grep “Random initial root password”
  • ssh root@$CIP
  • cd /var/www/gnusocial
  • vi config.php
  • change $config[‘site’][‘server’] (e.g. ‘mysocial.domain.com:4443’) & $config[‘site’][‘name’]
  • change desktop host file to reflect $config[‘site’][‘server’]
  • web browse to https://mysocial.domain.com:4443

et voilà.

For every mistake I made, I ran:

docker stop social
docker rm $(docker ps -a -q) # deletes all containers!

and started again.  You could do this in less than 15 minutes!

Posted in ITMS, Linux SysAdmin, social media, Virtual Machines | Tagged , | Leave a comment

Protected: An app idea

This content is password protected. To view it please enter your password below:

Posted in ITMS, Student Enterprise Movement, university | Tagged , , , , | Enter your password to view comments.

Test Pen

[iframe src=”http://panopto.dmu.ac.uk/Panopto/Pages/Embed.aspx?id=cbf51e90-266e-42ef-82fa-f56bebedaa4f&v=1″ width=”640″ height=”360″ frameborder=”0″]

Posted in Uncategorized | Leave a comment

Raspberry Pi 3 and HostAPD throughput

Raspberry Pi 3 in a Pi2Go

Raspberry Pi 3 in a Pi2Go

The Raspberry Pi 3 has n-wireless (and bluetooth) onboard now.  I got this robot before Christmas “for my boys”.  I didn’t have much time with it before the holidays and couldn’t quite get it to work with a USB wifi dongle and USB storage (to hopefully store pictures from the camera.)

Anyway, I got the Raspberry Pi 3 (RPi3) yesterday and got hostapd working in a very short time using this RPI Wireless Hotspot.

I posted on Twitter that I got it working and got this tweet in reply:

So, here goes… I installed iperf on the RPi3 and locally on my laptop (Fedora 22 i7) with RT5370 USB dongle and Intel 7260 AC onboard.

Raspberry Pi 3 Aerial

Raspberry Pi 3 Aerial (picture from http://makezine.com/2016/02/28/meet-the-new-raspberry-pi-3/)

I can’t find the reference on the web but someone asked if an external aerial could be fitted because his Pis are used in industrial applications and need to be housed in metal cases.  This got me to think that there might be a problem with a robot like mine when it rotates.

This is using the USB dongle on the laptop with the aerial facing away at two feet:

root@robotpi:/home/pi# iperf -s
[root@qedu ~]# iperf -c 10.1.0.1
------------------------------------------------------------
Client connecting to 10.1.0.1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.1.0.192 port 59856 connected with 10.1.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 24.5 MBytes 20.5 Mbits/sec

This is using the USB dongle on the laptop with the aerial facing towards it:

[root@qedu ~]# iperf -c 10.1.0.1
------------------------------------------------------------
Client connecting to 10.1.0.1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.1.0.192 port 60514 connected with 10.1.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 24.1 MBytes 20.1 Mbits/sec

From ten feet with the aerial facing away from the laptop:

[root@qedu ~]# iperf -c 10.1.0.1
------------------------------------------------------------
Client connecting to 10.1.0.1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.1.0.192 port 60792 connected with 10.1.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 24.0 MBytes 20.1 Mbits/sec

Now, with the AC wireless:

[root@qedu ~]# iperf -c 10.1.0.1
------------------------------------------------------------
Client connecting to 10.1.0.1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.1.0.153 port 56582 connected with 10.1.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.3 sec 20.4 MBytes 16.7 Mbits/sec
[root@qedu ~]# iperf -c 10.1.0.1
------------------------------------------------------------
Client connecting to 10.1.0.1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.1.0.153 port 56600 connected with 10.1.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 21.9 MBytes 18.2 Mbits/sec

Ouch!  The USB wireless-n dongle is faster than the onboard wireless-AC.

 

Posted in Linux SysAdmin, Raspberry Pi | Tagged , , , , , | 1 Comment

Protected: 2016 CentOS 7 virtual machine template to support LAMP and other applications. Part2

This content is password protected. To view it please enter your password below:

Posted in ITMS, Linux SysAdmin, Virtual Machines | Tagged , , , , , , | Enter your password to view comments.

Research Data Management: Hydra at the University of Hull

 Visiting University of Hull Library

The University of Hull has refurbished its library.  This is a graphical representation of it.

The University of Hull has refurbished its library. This is a graphical representation of it.

The University of Hull has refurbished its library at a cost of £28m and very nice it is too.

In the library, there is a lot of space, light and high ceilings.  General computing includes iiyama monitors who have a reputation (with geeks) as being a little less off-the-shelf than, for example, DELL.  Maybe the PCs are not not an afterthought to the service either.  I did not see it but there is an art gallery too.

The library has chairs that you might call privacy booths with high sides and a high back that you can sit at with a book or a tablet because of the convenient power sockets.  Eduroam wireless networking exists throughout.  There are student group meeting rooms that are not locked.  There are boardroom style meeting rooms with a mixture of Crestron based AV and old fashioned green leather clad configurable board tables.  The mixture works well, very “library”.

We, Alan Brine and I, were treated to lunch, which was nice and unexpected.  We spent the day with Chris Awre, Head of Information Management, and Richard Green, consultant: Hydra in Hull and related projects.

Hydra and the reason for the visit

Hydra About Hydra Introduction to Hydra

We are looking at developing a service around the curation of digital research data.  We already have DORA, which is our open repository for research output: publications and doctoral theses, which is just over 100GB.  We are used to DSpace and could create a open access data repository using it or even partition DORA somehow and store research data in that but data has the distinction that it can be re-used, re-purposed and, better, parts of different collections could be mashed by anyone who discovers the data.  This is what Hydra may deliver.

Before the visit we, in a rush, passed on these questions:

  • People:
    How many staff are involved in running the service?
    What do they do?
    What is the work flow leading up to creating a new collection?
    Has anyone received external training? In what?
  • Hardware:
    What is size of the archive so far?
    What is the expected growth?
    What are the specifics of the hardware solution?
  • Technical:
    What are the steps related to the software/creating new functionality?
    Are there any yearly costs for support or licensing? PURLs etc.

And while the conversation was freer than the structure above I hope to capture some of the answers here.

The Conversation

The first point I took the time to note was about the company Data Curation Experts who are important in two respects.  They are very much aware of Hydra but also could kick start any effort because of their knowledge around structured and linked data.  It is all very well to be a good custodian of a data repository but the value is in the factoring of data and its re-use.  That is something we need to implement and where we might have a gap in our knowledge and experience.  We need to understand and replicate as many patterns as possible so that others at the university and beyond can dip in to our repository with out too much of a learning curve.  Where possible we might make the effort to make the data machine-readable.

While DMU compared EPrints and DSpace and chose DSpace because of its worldwide community, Hull chose Hydra for similar reasons with the added essential that they always wanted to store data collections.  They started looking at Hydra in August 2011 with a developer working on it full time until 2013.  The developer learnt the programming language Ruby from scratch.  Today, Hull library has access to a developer in their equivalent of our ITMS.  We talked about training and discussed the active community.  In terms of training, Data Curation Experts can help but there is also Hydra Europe and Hydra Camp.  We talked about who has been involved with Hydra at Hull and how it works today.  Chris and Richard have been doing research curation at Hull for ten years plus.  The library has a team called Researcher Services which is partly comprised of their cataloguers.

The conversation moved here and there, some of it was about how Hull are doing library services.  I am a little envious at their facilities and use of Blacklight as the catalogue search frontend to their library management system.  Envious too, because they are being trusted to use, even embrace, open source software to provide university wide services.  Their previous exam paper collection is stored using Hydra as are undergraduate and graduate theses.  We do not currently store undergraduate theses but it seems like a bold initiative.  There are 11,000 objects in their repository and we are a smidge behind at 10,500.

We talked about workflow and the woes of the librarians herding academics; some are keen, some do not know about the service and some have the default position, often right, that their work can not go in to a open repository.  We talked about advocacy and support in governance from our universities.  At Hull, Hydra is seen by the university as infrastructure rather than an end point for dumping research data.  We talked about tools:

  • The Avalon Media System A next generation Hydra Head for Audio and Video delivery
  • Capistrano  A remote server automation and deployment tool written in Ruby
  • Archivematica is a web- and standards-based, open-source application which allows your institution to preserve long-term access to trustworthy, authentic and reliable digital content
  • Sufia is a Rails engine for creating a self-deposit institutional repository
  • Worthwhile is a simple institutional repository for Hydra
  • Hydra in a Box The Digital Public Library of America (DPLA), Stanford University and DuraSpace are partnering to extend the existing Hydra project codebase and its vibrant and growing community to build, bundle, and promote a feature-rich, robust, flexible digital repository that is easy to install, configure, and maintain.

The most important function of the tools above is exampled by Archivematica.  The tool deconstructs datasets, for example, Zip files and and prepares data for long term access.  However, the function is fraught with problems and difficult design decisions.  If we take the example of a Microsoft Excel spreadsheet.  The tools to access that spreadsheet may not always be available because of the proprietary nature of the product but even today we can not convert from Excel to an open format like the OpenOffice spreadsheet format because the tools that exist today can lose information in the Excel spreadsheet when converting to something more reliable.  There is a need for strong governance around what we store and to what formats we convert non-open data formats.  While talking about Archivematica, Richard showed us a workflow that he is looking at:

A possible workflow for Research Data Management, Richard Green. Creative Commons : CC-BY-NC-SA

A possible workflow for Research Data Management, Copyright Richard Green. Creative Commons : CC-BY-NC-SA

Long term preservation of research data is implemented through the use of Archivematica.  Other data that has been through the system before the introduction of the tool can be sent through again.  Data that is now seen as long-term rather than medium-term can also go through the tool.  The view on what format should be used for a long term preservation may change and that is catered for as collections may be re-ingested.

We are starting from scratch, we might like to see Archivematica or similar used for every collection.  This way researchers will access data collections in a similar way every time.

Conclusions

We could take an easier route than Hydra.  We could provide a repository where we make open access datasets discoverable and downloadable.  This approach would tick a box.  We would, however, have no idea what the data is being used for or how many times it has been used.  We would not know which aspects of the data are being used and what, therefore, is useful or has value.  The use of the data is separated from our systems and we can no longer track it.

We can do better than that for the research community.  Other organisations are looking for insight and value from data over and above its original purpose.  We can provide an infrastructure as a service IaaS based around Hydra or similar.  If data is organised in a way that researchers can access facets of it then we will have a better idea of what is being used and how often.  If we are talking about open access to research data as infrastructure then we must be talking about open source software.  If we are to provide an IaaS then anyone using it must be able to use it in a way that is easily reproducible and at as little cost as possible without any lock-in.  Good open source software and the communities around it supports this behaviour by default.  Proprietary software is based on the model of selling and reselling.  The business model includes practices such as built-in obsolesence.  This practice and others like it do not fit with open access to long term curation of data.

Another reason to go the IaaS route is the concern around privacy, intellectual property and anonymity.  This may answer the cloud question too.  There will be data that needs to have restricted access and with a need to be archived but sometimes it will be stored with other interesting data without the same restriction.  With data sets broken up and stored in facets we can control who has access to what.  We can control access with a straight forward repository too, but data that could be released would be hidden as part of a restricted blob of data.

If we implement the storage of research data (even data from other systems) as IaaS then we can create good practices and governance at the university that researchers and others can use.  We can advocate the use of open formats in order to avoid the problem explained above.  Implementing a RDM IaaS will provide the opportunity to reduce the number of systems we need to understand when working with researchers from induction, through maintaining their work and preservation of it.

I have written about this service being IaaS instead of an end point dump for data,  I have talked about the benefits of open source and open standards in general but what of Hydra?  Is it the right tool?  Back in 2010, the risk would be considerable.  Ruby on Rails at the time, while exciting, was considered limited.  The thinking was, that at some point the developer would reach the wall of what is possible and would have to start writing code at a lower entry point in the Ruby stack.  Hydra has been in use for at least five years.  The community is thriving, Ruby and Rails are thriving and Hydra as well as being developed as infrastructure is also being developed as a turnkey solution, for example Hydra in a Box.  Now, would certainly be a better time to enter the community than five years ago.

Postscript

For many years, oblivious to Hydra, I have wanted to work with Fedora Commons.  Hydra uses Fedora Commons.  Fedora Commons allows for the storage of digital collections using Triples to describe relationships.  Triples are used in “Big Data”.  While some of the hoopla around Big Data has died down an understanding about the need to reliably, with consistency, store data and data about data has come about.  Organisations are re-tooling to include Big Data tools in their infrastructure.  In the same period that Hydra has been maturing, I have learnt that DSpace would run on top of Fedora Commons but missed the opportunity to use it.  Since then I have become aware of Hadoop which promises horizontal scaling and secure storage of data with the added benefit of compute at the node the data sits on.  In looking at Hydra, I have come up with the idea that Hadoop could underpin Fedora Commons which in turn is a key component of Hydra.  These three layers are infrastructure most of the time.  They each expose web services which could be leveraged by programmers and existing tools.  We could sit DSpace on Fedora Commons and then have Hydra sit next to it or replace it.  Tools could sit on any layer to support research or the business functions of the university.  The framework would be very flexible.  DMU is looking at storage solutions at the moment and providers are telling us that they have Hadoop offerings.  Now could be the time.  Hadoop is ten years old and growing rapidly.  If we had Hadoop we might be able use it for teaching too.  The world, it seems, needs data curation and data scientists.

Posted in DORA, DSpace, DSpace, ITMS, Library, Linux SysAdmin, Uncategorized | Tagged , , , , , , , | Leave a comment