So I'm in a very odd predicament. I have a Dell Server with the main OS as Proxmox VE 5.3, and with a Nginx VM plus 3 other VMs. The Nginx VM's job is to Reverse Proxy the other 3 VMs through the one Public IP Address. I also have HTTPS set up for the 3 VMs on the Nginx VM (Certbot). Took a long number of months for me to set up last year, and it generally works great!
Except, about once a month or more, I will run into Disk Space issues on the Nginx VM. It will just magically max out from 78% to 100%. When it maxes out on disk space (or even around 96%), Nginx will stop working, and me (and my dev team) will be unable to connect to the other 3 VMs relying on Nginx Reverse Proxying. How do I resolve this issue usually? I have some manual methods:
1. I have a Proxmox Snapshot I keep of a working, non-maxed out Disk Space Nginx VM from 1 week and even a 2 month old one. I roll back to one of these (otherwise I can't access the Nginx VM).
2. I will renew the Certbot Licenses (I have sometimes noticed this will help with disk space, although it is set to auto-renew)
3. I go about trying to lower the disk space as much as a I can (see later below).
4. Stop the other 3 VMs and start them back up. After a few minutes they will start working again.
So what do I have assigned to the Nginx VM? I have the following:
Memory: 2 GB (Minimum), 3 GB (Maximum)
Processor: 2 (2 sockets, 1 cores)
Hard Disk (scsi0): local-lvm: vm-100-disk-0,size=10G
Now I know there is the obvious suggestion of "Assign the Nginx VM more Disk Space", but I have several reasons why I don't wish to:
1. I have a Mirrored Setup. Once I assign Disk Space to the Nginx VM, I can't deallocate the Disk Space. I would rather the extra unassigned Disk Space be saved for our Git Server, where our actual content is being saved to.
2. From everything I've been reading, 10 GB is actually a lot already assigned to an Nginx VM.
What have I do to try to alleviate the Disk Space issues so far? I run the following commands manually (based on some online guides I've read):
$ sudo apt-get autoremove (Auto Remove Applicaitons, usually this does nothing)
$ sudo apt-get autoclean (Auto Clean the Cache, sometimes there is some data to clean up)
$ sudo apt-get clean (Clear the whole cache, usually the most effective, but will only clean up maybe 2% of the 3 GB /dev/mapper/ubuntu--vg-ubuntu--lv)
$ sudo journalctl --vacuum-time=60d (Clear journals older than 60 Days, sometimes cleans up a few MB)
$ rm -rf ~/.cache/thumbnails/* (Clean the Thumbnail Cache; I don't have a GUI, so I doubt this is doing anything)
When it starts filling up mysteriously (in the matter of a couple minutes), I will get the following messages:
E: Could not get lock /var/cache/apt/archives/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the download directory
Do you have any recommendations for how I can keep Nginx from filling up? The Disk Space of /dev/mapper/ubuntu--vg-ubuntu--lv (I used Ubuntu Server as the base OS) is usually sitting around 78-80%, and like I mentioned, will just mysteriously start to fill up and magically max out, requiring me to Snapshot Rollback to a previous point when it was NOT maxed out.
Except, about once a month or more, I will run into Disk Space issues on the Nginx VM. It will just magically max out from 78% to 100%. When it maxes out on disk space (or even around 96%), Nginx will stop working, and me (and my dev team) will be unable to connect to the other 3 VMs relying on Nginx Reverse Proxying. How do I resolve this issue usually? I have some manual methods:
1. I have a Proxmox Snapshot I keep of a working, non-maxed out Disk Space Nginx VM from 1 week and even a 2 month old one. I roll back to one of these (otherwise I can't access the Nginx VM).
2. I will renew the Certbot Licenses (I have sometimes noticed this will help with disk space, although it is set to auto-renew)
3. I go about trying to lower the disk space as much as a I can (see later below).
4. Stop the other 3 VMs and start them back up. After a few minutes they will start working again.
So what do I have assigned to the Nginx VM? I have the following:
Memory: 2 GB (Minimum), 3 GB (Maximum)
Processor: 2 (2 sockets, 1 cores)
Hard Disk (scsi0): local-lvm: vm-100-disk-0,size=10G
Now I know there is the obvious suggestion of "Assign the Nginx VM more Disk Space", but I have several reasons why I don't wish to:
1. I have a Mirrored Setup. Once I assign Disk Space to the Nginx VM, I can't deallocate the Disk Space. I would rather the extra unassigned Disk Space be saved for our Git Server, where our actual content is being saved to.
2. From everything I've been reading, 10 GB is actually a lot already assigned to an Nginx VM.
What have I do to try to alleviate the Disk Space issues so far? I run the following commands manually (based on some online guides I've read):
$ sudo apt-get autoremove (Auto Remove Applicaitons, usually this does nothing)
$ sudo apt-get autoclean (Auto Clean the Cache, sometimes there is some data to clean up)
$ sudo apt-get clean (Clear the whole cache, usually the most effective, but will only clean up maybe 2% of the 3 GB /dev/mapper/ubuntu--vg-ubuntu--lv)
$ sudo journalctl --vacuum-time=60d (Clear journals older than 60 Days, sometimes cleans up a few MB)
$ rm -rf ~/.cache/thumbnails/* (Clean the Thumbnail Cache; I don't have a GUI, so I doubt this is doing anything)
When it starts filling up mysteriously (in the matter of a couple minutes), I will get the following messages:
E: Could not get lock /var/cache/apt/archives/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the download directory
Do you have any recommendations for how I can keep Nginx from filling up? The Disk Space of /dev/mapper/ubuntu--vg-ubuntu--lv (I used Ubuntu Server as the base OS) is usually sitting around 78-80%, and like I mentioned, will just mysteriously start to fill up and magically max out, requiring me to Snapshot Rollback to a previous point when it was NOT maxed out.