Nginx Reverse Proxy Invalid Security Token

I have a recent install of Grav, all plugins are up to date and running on Ubuntu 18.04/PHP 7.2/Apache.

Everything works fine browsing the site/admin panel on my LAN by IP, but if I use my URL either on LAN or WAN and try to log into the admin panel I get a red bar that says "Invalid Security Token". A few pages have ACL, and if I try to login there I get an access denied message, again only over the URL.

The website is behind a Nginx reverse proxy that is also handling the SSL, and I'm pretty sure this is my issue... Nginx is just doing a proxy_pass to the IP:port of the Grav server. I'm just not sure where to start with the troubleshooting. I cannot find anything in the logs showing what the issue may be.

I've tried:
Secure to No
Reverse Proxy to Yes
Remote Verify Peer to Off

Have you tried adding a location .../admin to the nginx.conf file?

   ## Begin - Index
    # for subfolders, simply adjust:
    # `location /subfolder {`
    # and the rewrite to use `/subfolder/index.php`
    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }
    ## End - Index

While troubleshooting I ended up swapping the ports Grav and Nextcloud were running on, and it worked! So I thought maybe I need to have Grav on port 80 and then swapped my Nginx config around so the subdomains were correct and it quit working again... The only common variable is that my Nginx config for the Nextcloud subdomain proxy had:

proxy_pass http://ipaddress:port;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;

Where-as the Grav config was simply:

proxy_pass http://ipaddress:port;

After adding the proxy_set_headers to the config for the Grav site it is working great.