How to make #JBoss 7.4+ a cluster ready service using #bash scripts and #API calls

JBoss applications server is a very nice and powerful Java application server from RedHat.

To make it really useful in a modern architecture of a complex software product we must prepare JBoss to be very easily deployed in a cluster environment.

The simplest way to do it is to write some basic Bash scripts that use the JBoss API and implement the most basic control functions needed by a cluster orchestrator to control the service.

This basic functions are:

• A start function that will start our service
• A stop function that will graciously stop the service
• A health check function that will be used by the cluster orchestrator to asses the health of the service and decide if the service needs to be restarted
• A status check function that will be used by the orchestrator to asses the detailed status of the service and decide if additional action should be taken: recovery , scaling the service to additional instances , downscaling the service instances etc.

Start function

To start the Jboss instance we can use the following script appStart.sh:

#!/bin/bash
source $(dirname "$0")/config.sh

function runConditionalApp {
        STATUS=`./appHealth.sh`
        if [ $STATUS == 'UP' ] || [ $STATUS == 'DEGRADED' ]
        then
                echo App is already running with STATUS $STATUS
        else
                echo "Starting " $APP_DOMAIN
                rm -rf $APP_DOMAIN/standalone/configuration/standalone_xml_history/current
                $APP_DOMAIN/bin/standalone.sh -b 0.0.0.0 -bmanagement 0.0.0.0 --debug 8790 &> /dev/null &
                sleep 10
        fi
}
runConditionalApp

Notes:

• config.sh declares the environment variables like $APP_DOMAIN that indicate where our JBoss is installed.
• the whole start method uses the standard way to start JBoss application server as a standalone instance.
• note that the conditional used to check if we can start is based on the health check script appHealth.sh
• http://localhost:9993 is the address of the JBoss administration console.

Stop function

To stop the Jboss instance we can use the following script appStop.sh:

#!/bin/bash
source $(dirname "$0")/config.sh

echo "Killing...  " $APP_DOMAIN
kill -9 $(jps -v | grep $APP_DOMAIN | awk '{print $1}')

Notes:

• config.sh declares the environment variables like $APP_DOMAIN that indicate where our JBoss is installed.
• the whole stop method uses kill to kill the whole process.
• note that there is usually a graceful stop implemented by JBoss but in a cluster environment where we may need a quick stop killing the process is more efficient. This has the added benefit of being able to stop also locked instances that will not stop gracefully.

Status function

To check the status of JBoss we have a very useful and already available API call stating with Jboss 7.4 version. The following bash script appStatus.sh can be used to return the status.

#!/bin/bash

curl -s -X GET http://localhost:9993/health -H 'cache-control: no-cache' | python -m json.tool 2> /dev/null

Notes:

• curl is used to call the API endpoint
• python is used to beautify the output json
• status information will be available in a json format that can be further processed by the cluster orchestrator and make decisions based on the reported information.
• http://localhost:9993 is the address of the JBoss administration console.

Health Check function

To check the health of the service we use the above status API call but we reduce the status information to a simple selection of health statuses: UP/DOWN/DEGRADED

The following script appHealth.sh implements that.

#!/bin/bash

LIST=$(curl -s -X GET http://localhost:9993/health -H 'cache-control: no-cache' \
| python -m json.tool 2> /dev/null \
| grep '\"outcome\"' \
| cut -d ':' -f 2 \
| sed 's/"/\"/g')

if  [ -z "$LIST" ]; then
    echo "DOWN"
else
    UP="true"
    for value in $LIST
    do
        if [ $value != "true" ] && [ $value != "true," ]; then
            UP="false"
        fi
    done
    if [ $UP == "true" ]; then
        echo "UP"
    else
        echo "DEGRADED"
    fi
fi

Notes:

• curl is used to call the API endpoint
• http://localhost:9993 is the address of the JBoss administration console.

Update:

In case python is not available on the platform the script has to run the following is a grep only variant.

#!/bin/bash

LIST=$(curl -s -X GET http://localhost:9993/health -H 'cache-control: no-cache' \
| grep -o '"outcome" : [^},]*' \
| cut -d ':' -f 2 \
| sed 's/"/\"/g')

if  [ -z "$LIST" ]; then
    echo "DOWN"
else
    UP="true"
    for value in $LIST
    do
        if [ $value != "true" ] && [ $value != "true," ]; then
            UP="false"
        fi
    done
    if [ $UP == "true" ]; then
        echo "UP"
    else
        echo "DEGRADED"
    fi
fi