Supervisor: it keeps your app running

Check status

service supervisor status

Tail a service's err

(find the file by looking in folder: /etc/supervisor/conf.d where file pdf.nimbletext.com.conf might mention error log /var/log/pdf.nimbletext.err.log)

sudo tail -n 20 -f /var/log/pdf.nimbletext.err.log

Restart supervisor

service supervisor stop
service supervisor start

supervisor is a nice thing. It supervises your app to keep it running.

sudo apt-get update
sudo apt-get install supervisor

Config goes in...

cd /etc/supervisor/conf.d/

To create a new service for supervisor to supervise, add a file, e.g. helloworld.conf in the directory mentioned above, with content such as

$ cat helloworld.conf
[program:helloworld]
command=/usr/bin/dotnet /home/user1/publish/MvcMovie.dll
#var/aspnetcore/HelloMVC/HelloMVC.dll
#directory=/var/aspnetcore/HelloMVC/
directory=/home/user1/publish/
autostart=true
autorestart=true
stderr_logfile=/var/log/helloworld.err.log
stdout_logfile=/var/log/helloworld.out.log
environment=ASPNETCORE_ENVIRONMENT=Production
user=www-data
stopsignal=INT

View supervisor logs

Here's how to view the last 20 lines of the supervisor log, using tail

sudo tail -n 20 -f /var/log/supervisor/supervisord.log

...in which you might see it struggle to keep your app running...

Or the last 20 lines of your app's error output...

sudo tail -n 20 -f /var/log/helloworld.err.log

Or the last 20 lines of your app's standard (non error) output...

sudo tail -n 20 -f /var/log/helloworld.out.log

Restart supervisor - troubleshooting

To restart supervisor after you've given it some new configuration:

service supervisor stop
service supervisor start

I noticed that supervisor did not successfully restart when I rebooted my linux box.

I restarted it with service supervisor start (which required me to type my password)

First though, I checked the status with:

service supervisor status

and saw:

user1@machine1:~$ service supervisor status
? supervisor.service - Supervisor process control system for UNIX
	 Loaded: loaded (/lib/systemd/system/supervisor.service; disabled; vendor preset: enabled)
	 Active: inactive (dead)
		 Docs: http://supervisord.org

Starting it went like this...

user1@machine1:~$ service supervisor start
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to start 'supervisor.service'.
Authenticating as: user1,,, (user1)
Password:
==== AUTHENTICATION COMPLETE ===

Check status of supervisor

And now I get this result from status --

user1@machine1:~$ service supervisor status
? supervisor.service - Supervisor process control system for UNIX
	 Loaded: loaded (/lib/systemd/system/supervisor.service; disabled; vendor preset: enabled)
	 Active: active (running) since Thu 2017-04-06 07:19:51 EDT; 4min 19s ago
		 Docs: http://supervisord.org
 Main PID: 2795 (supervisord)
		Tasks: 41
	 Memory: 275.4M
			CPU: 24.093s
	 CGroup: /system.slice/supervisor.service
					 +-2795 /usr/bin/python /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf
					 +-2800 /usr/bin/dotnet /opt/webapps/webapp1/app/app1.dll
					 +-2801 /usr/bin/dotnet /opt/webapps/webapp2/app/app2.dll
					 +-2802 /usr/bin/dotnet /opt/webapps/webapp3/app/app3.dll

Another way to inspect the status of supervisor:

systemctl status supervisor.service

Exactly same output as service supervisor status...e.g.:

? supervisor.service - Supervisor process control system for UNIX
 Loaded: loaded (/lib/systemd/system/supervisor.service; disabled; vendor preset: enabled)
 Active: active (running) since Mon 2017-09-11 02:00:44 UTC; 1min 27s ago
	 Docs: http://supervisord.org
 Main PID: 6654 (supervisord)
	Tasks: 41
	 Memory: 207.6M
		CPU: 14.514s
   CGroup: /system.slice/supervisor.service
		 +-6654 /usr/bin/python /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf
		 +-6659 /usr/bin/dotnet /opt/webapps/yourfirstproduct.com/app/YourFirstProduct.dll
		 +-6660 /usr/bin/dotnet /opt/webapps/codesmackdown.com/codesmackdown.com.dll
		 +-6661 /usr/bin/dotnet /opt/webapps/joy.nimbletext.com/app/Joy.dll

Make supervisor restart automatically

To restart supervisor on reboot, add something like this to your crontab:

@reboot	/my/path/to/supervisord -c /my/path/to/supervisord.conf 2>&1

i.e.

@reboot /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf 2>&1

Add that to your cron jobs using crontab (described in cron).

Useful .ok file for /etc/supervisor/conf.d folder

I use ok.sh on all my linux/mac systems (and ok-ps on all my windows systems)

ls *.conf -al # view config of each app
cat *.conf  | grep 'out\.log\|program'
sudo tail -n 100 -f /var/log/supervisor/supervisord.log # check sup's log
sudo tail -n 150 -f /var/log/myapp1.out.log
sudo tail -n 150 -f /var/log/myapp2.out.log
service supervisor status
cd /etc/nginx/; d # nginx

(Those lines that say for example: sudo tail -n 150 -f /var/log/myapp1.out.log ...i have something other than myapp1 -- i have the name of the log files output by my web apps. The same files I can find the name of with cat *.conf | grep 'out\.log\|program')

The last line of the .ok file takes me to the nginx config folder, where I have another .ok file.

Sources

See also