Troubleshooting ghost post scheduling failures

by Derek Swingley

Earlier this week I upgraded ghost and finally got on a release that can schedule posts. Or so I thought. The UI to let me schedule posts was right there– but every test I did failed to publish my test post at the scheduled time. Instead, I would see "scheduled" in the admin UI and the post would never publish. Interestingly, if I edited the post, it would immediately show.

I mentioned this to a friend who runs ghost as well and basically got "dunno, womm". He did suggest I try the ghost slack so I headed over there.

I exchanged a few messages with a ghost dev but we didn't really get anywhere (we were thinking maybe it was a timezone issue but that ended up not being the case).

I kept poking around and looked up and down the stack:

  • watching network traffic from my browser to ghost
  • looking at ghost's logs, which are basically nginx's logs
  • comparing entries in my instance of ghost's DB with stuff from a fresh 0.10.1 version of ghost

I learned a little about how ghost's post scheduling works in this process. I now know that behind the scenes ghost uses additional processes to make requests to its own API endpoint when it's time for a post to go from scheduled to published. In the logs, these look like:

PUT /ghost/api/v0.1/schedules/posts/7?client_id=ghost-scheduler&client_secret=9045df2f1b06 200 74.665 ms - 525

Not really knowing anything about ghost-scheduler, I searched the ghost repo. Those search results lead me to publishPost in server/api/schedules.js and then edit in server/api/posts.js. A few console.log statements later I knew what was coming through those functions in my local instance of ghost when a post was successfully scheduled and then published. Doing the same thing on my production server I could see that the same functions weren't running as a result of similar requests to /posts with client_id=ghost-scheduler.

I was comparing my local copy of ghost with the busted one on my server, comparing logs and I noticed that the requests made by ghost-scheduler on my server were off. For instance:

GET /ghost/api/v0.1/schedules/posts/49?client_id=ghost-scheduler&client_secret=dc49d737bfd2 HTTP/1.1" 301 146 "-" "node-superagent/2.2.0

See it? That's a GET instead of a PUT. I knew that wasn't right, since watching the traffic from successfully scheduled posts on a local instance of ghost always did PUTs to change a post from scheduled to published. Since my server runs ghost behind nginx, I figured it had to be something related to my nginx configuration.

Side note: a couple of months ago I set up an SSL certificate for this site using let's encrypt. As part of that process, all requests to http get redirected to https.

I started researching why nginx would convert a PUT to a GET while reverse proxying requests. That was a dead end as I couldn't find anything that fit the behavior I was seeing. I found some info around rewriting URLs and http verb changing, but that's not what my nginx config was doing. My server returns a 301 for requests made over http with the same URL with the protocol changed to https.

At some point during all this tinkering I re-discovered the ghost config file and that it specifies a URL. Taking a look at config.js in the root of my ghost install I saw:

production: {  
    url: 'http://derekswingley.com/',
    <snip>

There we go! Clearly I was one s away from making everything work. If only. I changed to https, restarted ghost and my site wouldn't load because ERR_TOO_MANY_REDIRECTS. What the heck.

Luckily, I wasn't ghost user to have this problem. This time the fix was indeed an nginx configuration tweak. In my nginx config, in the block that does the reverse proxying, I added:

proxy_set_header X-Forwarded-Proto $scheme;

I hadn't seen the X-Forwarded-Proto header before but it's there to tell the receiver the protocol used to make the original request. Knowing the protocol, ghost was no longer redirecting to SSL since it was being told "hey, this already came from SSL" and was getting down to doing the work asked of it.

After making that change to nginx, I restarted nginx and what do you know, I can schedule posts!