Finally My Own AWX Server

Share on:
Finally My Own AWX Server

Finally, My Own AWX Server

I've been wanting my own AWX server for ages

I've messed with a bit of Ansible and found it to be agreeable with some of the previous ways Ive automated code in the past, but I wanted to scale it up (given my infrastucture roots).

I attempted this install two or three times this year across different platforms and install types and had problems sometimes being caused with me experimenting with other technologies at the same time or just out of date documentation.

Below is the instructions for my first working AWX installation.

First our dependencies and what we'll end up installing:

Software Version
Windows 10 Pro (Host) 21H2, 19044.2364
VirtualBox 6.1.32
Ubuntu 22.04.1 LTS
Docker CE 20.10.22
Minikube 1.28.0
Kubectl 1.26.0
Kubernetes 1.25.3
Kustomize 4.5.7
AWX 21.10.2

1. Get a Ubuntu VM up and running in VirtualBox.

Fairly simple this instruction, download the latest Ubuntu 22.04 LTS iso and install to a VM in Virtualbox. I gave the VM 8GB RAM, 4 CPUs and 40GB of disk space. I wanted enough room for docker to breathe inside the VM.

I picked Ubuntu as its an easy installation, has a nice GUI to fall back on if need be and well supported by its community.

I also made the network card bridged on to my network direct so my other machines could talk to it if need be.

2. Follow the instructions

In the main, I did follow the instructions that are here but had a couple of stumbling blocks so I thought I'd make it clear here.

2a. Install Docker CE

Refresh the apt caches for most up to date packages and install

1sudo apt update
2sudo apt upgrade

This is for my own personal preference so I can remote in from either my Windows or Mac.

1sudo apt-get install openssh-server

Install dependencies to add docker repositories

1sudo apt install apt-transport-https wget curl gnupg-agent ca-certificates software-properties-common -y

add the GPG keys to allow addition of apt repositories

1curl -fsSL | sudo apt-key add -

add the docker repositories into the apt lists

1sudo add-apt-repository "deb [arch=amd64] jammy stable"

docker install...

1sudo apt install docker-ce docker-ce-cli -y

make sure docker user group exists

1newgrp docker

add your user to the docker group (allows non-root users to mess with docker)

1sudo usermod -aG docker $USER

test out the docker install.

1docker version

Should show something like this on success: Docker Version results returning

2b. Install and setup minikube

Minikube allows a developer to run his own "mini" kubernetes cluster to get to know the software and mess with pods, etc.

You can run it "direct" on Windows, but I wanted to move away from that due to the difficulties I've had in the past, so I'm running minikube inside my Ubuntu VM.

Fortunately docker and minikube on Linux run differently to when run on Mac so I'm not doing full Virtualisation within a Linux VM (nested virtualisation). Not yet noticed too much of a performance penalty for doing this.

Download the minikube binary:


Copy the minikube binary into standard binaries folder

1sudo cp minikube-linux-amd64 /usr/local/bin/minikube

Give it execution permissions

1sudo chmod +x /usr/local/bin/minikube

make it easy to run kubectl commands inside minikube (It is not required for kubectl to be separately installed since it comes already wrapped inside minikube. As demonstrated above, simply prefix minikube kubectl -- before kubectl command, i.e. kubectl get nodes would become minikube kubectl -- get nodes)

1alias kubectl="minikube kubectl --"

...and test.

1minikube version

Should show something like this on success: Minikube Version results returning

Finally lets get the kubernetes cluster up and running (--addons=ingress very important here. This process didn't work without it).

1minikube start --cpus=4 --memory=6g --addons=ingress

Should see something similar to this:

 1$ minikube start --cpus=4 --memory=6g --addons=ingress
 2😄  minikube v1.28.0 on Ubuntu 22.04.1
 3✨  Using the docker driver based on existing profile
 4👍  Starting control plane node minikube in cluster minikube
 5🚜  Pulling base image ...
 6🏃  Updating the running docker "minikube" container ...
 7🐳  Preparing Kubernetes v1.25.3 on Docker 20.10.22 ...
 8🔎  Verifying Kubernetes components...
 9    ▪ Using image
10    ▪ Using image
11    ▪ Using image
12    ▪ Using image
13🔎  Verifying ingress addon...
14🌟  Enabled addons: storage-provisioner, default-storageclass, ingress
15🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Checking minikube's status, you should see this:

 1$ minikube kubectl -- get nodes
 2NAME       STATUS   ROLES                  AGE    VERSION
 3minikube   Ready    control-plane,master   113s   v1.28.0
 5$ minikube kubectl -- get pods -A
 6NAMESPACE       NAME                                        READY   STATUS      RESTARTS   AGE
 7ingress-nginx   ingress-nginx-admission-create--1-kk67h     0/1     Completed   0          2m1s
 8ingress-nginx   ingress-nginx-admission-patch--1-7mp2r      0/1     Completed   1          2m1s
 9ingress-nginx   ingress-nginx-controller-69bdbc4d57-bmwg8   1/1     Running     0          2m
10kube-system     coredns-78fcd69978-q7nmx                    1/1     Running     0          2m
11kube-system     etcd-minikube                               1/1     Running     0          2m12s
12kube-system     kube-apiserver-minikube                     1/1     Running     0          2m16s
13kube-system     kube-controller-manager-minikube            1/1     Running     0          2m12s
14kube-system     kube-proxy-5mmnw                            1/1     Running     0          2m1s
15kube-system     kube-scheduler-minikube                     1/1     Running     0          2m15s
16kube-system     storage-provisioner                         1/1     Running     0          2m11s

2c. Install Kustomize

Kustomize is a Kubernetes configuration transformation tool that enables you to customize untemplated YAML files, leaving the original files untouched.

1curl -s ""  | bash
3sudo mv ./kustomize /usr/local/bin

3. Create a kustomization.yaml file and deploy

Create a folder to put your yaml files

1mkdir awxPlay
3cd awxPlay

create kustomization.yaml in the awxPlay folder with the below contents. Replace <tag> with the latest tag on the AWX github site (I put 1.1.3)

 2kind: Kustomization
 4  # Find the latest tag here:
 5  -<tag>
 7# Set the image tags to match the git version from above
 9  - name:
10    newTag: <tag>
12# Specify a custom namespace in which to install AWX
13namespace: awx

Install the manifests by doing:

 1$ kustomize build . | kubectl apply -f -
 3namespace/awx created created created created
 7serviceaccount/awx-operator-controller-manager created created created created created created created created
15configmap/awx-operator-awx-manager-config created
16service/awx-operator-controller-manager-metrics-service created
17deployment.apps/awx-operator-controller-manager created

After some time, we should be able to check it with:

1$ kubectl get pods -n awx
3NAME                                               READY   STATUS    RESTARTS   AGE
4awx-operator-controller-manager-66ccd8f997-rhd4z   2/2     Running   0          11s

4. Create an AWX deployment file and add to kustomization.yaml

In your awxPlay folder, create a file called awx-demo.yaml with the following contents, I changed <nodeport_port> to 30080:

3kind: AWX
5  name: awx-demo
7  service_type: nodeport
8  # default nodeport_port is 30080
9  nodeport_port: <nodeport_port>

and reference that file in the kustomization.yaml like this:

3  -<tag>
4  # Add this extra line:
5  - awx-demo.yaml

...and finally run kustomize again:

1kustomize build . | kubectl apply -f -

If everything works, you should get a running AWX instance. It'd look like this: kubectl get pods after AWX install

5. Login and see it....

This command shows you the URL with which to access AWX instance.

1$ minikube service -n awx awx-demo-service --url

User is admin, password is magically revealed when you run the below.

1$ kubectl get secret awx-demo-admin-password -o jsonpath="{.data.password}" -n awx | base64 --decode ; echo

Appendix: Some of the Errors I saw along the way....

So below we see where I was getting issues with Pods starting.


 1$ kubectl get pods -A
 3  NAMESPACE       NAME                                               READY   STATUS                  RESTARTS         AGE
 4  awx             awx-demo-589886bb5-r6njx                           0/4     Init:ImagePullBackOff   0                69m
 5  awx             awx-demo-postgres-13-0                             1/1     Running                 0                70m
 6  awx             awx-operator-controller-manager-577f6968b5-qx9xl   1/2     CrashLoopBackOff        12 (2m27s ago)   71m
 7  ingress-nginx   ingress-nginx-admission-create-64tgt               0/1     Completed               0                160m
 8  ingress-nginx   ingress-nginx-admission-patch-4cz6h                0/1     Completed               0                160m
 9  ingress-nginx   ingress-nginx-controller-5959f988fd-99vdd          1/1     Running                 0                160m
10  kube-system     coredns-565d847f94-7cj6m                           1/1     Running                 0                160m
11  kube-system     etcd-minikube                                      1/1     Running                 0                160m
12  kube-system     kube-apiserver-minikube                            1/1     Running                 0                160m
13  kube-system     kube-controller-manager-minikube                   1/1     Running                 0                160m
14  kube-system     kube-proxy-79rqj                                   1/1     Running                 0                160m
15  kube-system     kube-scheduler-minikube                            1/1     Running                 0                160m
16  kube-system     storage-provisioner                                1/1     Running                 12 (2m26s ago)   160m

After much tracking back through logs and kubectl describes I got to the below workarounds


1  Type     Reason   Age                    From     Message
2  ----     ------   ----                   ----     -------
3  Warning  Failed   15m (x15 over 94m)     kubelet  Failed to pull image "": rpc error: code = Unknown desc = context deadline exceeded
4  Normal   BackOff  6m18s (x254 over 94m)  kubelet  Back-off pulling image ""
5  Warning  Failed   35s                    kubelet  Failed to pull image "": rpc error: code = Unknown desc = context deadline exceeded

This looks to relate image "" being large and not pulling quick enough, so we manually intervene and get it into our docker library by hand.


1$ minikube ssh docker pull


1  Type     Reason   Age                   From     Message
2  ----     ------   ----                  ----     -------
3  Warning  Failed   6m15s (x13 over 71m)  kubelet  Failed to pull image "": rpc error: code = Unknown desc = context deadline exceeded
4  Normal   BackOff  3m7s (x192 over 71m)  kubelet  Back-off pulling image ""

Same again, different image. This looks to relate image "" being large and not pulling quick enough, so we manually intervene and get it into our docker library by hand.


1$ minikube ssh docker pull