{"componentChunkName":"component---src-templates-blog-post-js","path":"/blog/publishing-screenshots-screencasts-and-other-files-on-digitalocean-spaces","result":{"data":{"allGhostPost":{"edges":[{"node":{"title":"Publishing Screenshots, Markdown Documents, and Other Files on DigitalOcean Spaces","html":"
\"screenshot\"
\n

A couple of weeks ago at DigitalOcean, we introduced Spaces—beautifully simple and reliable object storage. With Spaces, you can host and serve static files without having to worry about security, scaling, disk space, and all the fun things that come with maintaining a server.

\n

Spaces is compatible with the S3 API. This is important because it makes Spaces fully available to use with a huge number of applications and tools right away. In this post, I'll talk about how I'm using Dropshare to share screenshots, markdown files (rendered as HTML!), and all sorts of files on a custom domain name backed by Spaces.

\n

Features

\n

The most important part about this is that you are able to use your own domain name instead of bucket.region.digitaloceanspaces.com. But that's not all. In addition to simply serving static files, kmlnsr.me also renders markdown files as HTML, with the ability to view the raw Markdown source code. Take a look at this file for example. It's a Markdown file that is rendered as HTML with styling, on the fly. You can view the original Markdown code by appending /raw to the URL. Spaces does not support that natively, though. More on how that works in a bit.

\n

Setup

\n

Uploading files

\n
\"screenshot\"
\n

Dropshare uses the S3 API to upload files to Spaces. It supports custom S3 endpoints natively, so it's very easy to configure. Simply create a new connection with the following configuration:

\n
\"screenshot\"
\n

You might also want to replace the default screenshot hotkey to use Dropshare. First, disable the default hotkey in System Preferences→Keyboard→Shortcuts→Screen Shots. Then, configure Dropshare to use that shortcut in the Screenshots tab.

\n

For reference, here's Dropshare documentation on setting up DigitalOcean Spaces connections:

\n\n

Serving files

\n
\"screenshot\"
\n

Now this diagram is a bit more complex than the previous one, so let's break it down piece by piece.

\n
    \n
  1. \n

    A browser makes a request to a file on kmlnsr.me (like the screenshot above!). The request goes to a Droplet with nginx running on it.

    \n
    2. \n
  2. \n

    Nginx looks at the URL and checks if it is a Markdown file.

    \n

    a. If it is not a Markdown file, it proxies the request to Spaces, fetching the file and serving it as is to the browser.

    \n

    b. If it is a Markdown file, it fetches the file from Spaces, passes it to a markdown renderer, and then serves the HTML version to the browser.

    \n
    3. \n
\n

Nginx Configuration

\n

Let's build the configuration bit by bit, starting with a basic one:

\n
server {\n    listen 80;\n\n    server_name kmlnsr.me;\n    root /dev/null;\n    \n    location = / {\n        return 302 https://kamal.io;\n    }\n}\n
\n

Right now, it doesn't do much. http://kmlnsr.me/ redirects to https://kamal.io, and everything else returns a 404 Not Found error.

\n

Serving static files

\n

The idea behind serving static files is fairly simple. Proxy requests to https://kmlnsr.nyc3.digitaloceanspaces.com without any post-processing.

\n

Because we will set up a couple of location blocks, let's extract the common directives into their own file that we can simply include wherever we need it. I saved it in /etc/nginx/snippets/spaces_proxy.conf.

\n
proxy_http_version     1.1;\nproxy_set_header       Authorization '';\nproxy_hide_header      x-amz-id-2;\nproxy_hide_header      x-amz-request-id;\nproxy_hide_header      Set-Cookie;\nproxy_ignore_headers   "Set-Cookie";\nproxy_intercept_errors on;\n
\n

Now, let's serve some static files! location / matches all requests—except / itself as it is matched by a previous location block.

\n
location / {\n    include snippets/spaces_proxy.conf;\n    \n    proxy_set_header Host kmlnsr.nyc3.digitaloceanspaces.com;\n    proxy_pass https://kmlnsr.nyc3.digitaloceanspaces.com;\n}\n
\n

The proxy_set_header directive is important. Without it, Spaces would receive kmlnsr.me as the Host and wouldn't know what bucket it is, as it doesn't understand custom domain names.

\n

With this config, everything should work as if you are accessing https://YOUR_SPACE.nyc3.digitaloceanspaces.com directly.

\n

Serving Markdown files

\n

There are two parts to this feature:

\n
    \n
  1. Serving the raw Markdown file as-is
    2. \n
  2. Taking that, rendering it, and serving it as HTML
    3. \n
\n

Serving raw Markdown

\n

Right now, if you browse to /markdown_file.md/raw, you would get a 404 Not Found error. This is expected, because you are trying to access a file named raw in the directory /markdown_file.md.

\n

In order to fix that, we need to modify the URL that is passed to Spaces, removing the /raw bit. This can easily be done with a bit of RegEx magic.

\n
location ~ ^/(.*)\\.md/raw$ {\n    include snippets/spaces_proxy.conf;\n    \n    add_header Content-Type text/plain;\n    proxy_set_header Host kmlnsr.nyc3.digitaloceanspaces.com;\n    proxy_pass https://kmlnsr.nyc3.digitaloceanspaces.com/$1.md;\n}\n
\n

The URI pattern ^/(.*)\\.md/raw$ matches requests in the format of */markdown_file.md/raw, capturing the path to the Markdown file in the process. Once we have that, we simply set the URL that is sent to Spaces to */markdown_file.md.

\n

This way, we get the file's content, and serve it back to the browser. Setting the Content-Type header to text/plain asks browsers to display the file as plain text instead of downloading it.

\n

Rendering Markdown as beautiful HTML

\n

I've had this set up for years, so it uses a simple PHP script to do the hard work. You can download the source code here, hosted on Spaces of course :)

\n

In that archive, you will find two files:

\n
    \n
  1. markdown.php this is the Markdown library that I'm using. It's a very old version, but it has been working well so why bother replacing it ¯\\_(ツ)_/¯
    2. \n
  2. md.php this is the file that Nginx runs. It's not super complex, but it basically requests the source (using the /raw URL), parses it using the Markdown library, and serves it along with a basic CSS stylesheet. Unfortunately, I don't remember where I got the stylesheet from.
    3. \n
\n

You will need to adjust a couple of things in md.php. First, set the $url variable (line 5) to your own hostname. Then, copy the stylesheet to your Space and update the URL on line 21.

\n
location ~ \\.md$ {\n    fastcgi_pass    unix:/var/run/php5-fpm.sock;\n    fastcgi_param   DOCUMENT_ROOT    $document_root;\n    fastcgi_param   SCRIPT_NAME      $uri;\n    fastcgi_param   SCRIPT_FILENAME  /srv/markdown/md.php;\n    include fastcgi_params;\n}\n
\n

This is a basic php-fpm configuration. The most important part is that SCRIPT_FILENAME is fixed and always set to the md.php handler.

\n

Putting it all together

\n

Once you're all done, the config file will look like this:

\n
server {\n    listen 80;\n\n    server_name kmlnsr.me;\n    root /dev/null;\n    \n    location = / {\n        return 302 https://kamal.io;\n    }\n    \n    location / {\n        include snippets/spaces_proxy.conf;\n    \n        proxy_set_header Host kmlnsr.nyc3.digitaloceanspaces.com;\n        proxy_pass https://kmlnsr.nyc3.digitaloceanspaces.com;\n    }\n\n    location ~ ^/(.*)\\.md/raw$ {\n        include snippets/spaces_proxy.conf;\n    \n        add_header Content-Type text/plain;\n        proxy_set_header Host kmlnsr.nyc3.digitaloceanspaces.com;\n        proxy_pass https://kmlnsr.nyc3.digitaloceanspaces.com/$1.md;\n    }\n\n    location ~ \\.md$ {\n        fastcgi_pass    unix:/var/run/php5-fpm.sock;\n        fastcgi_param   DOCUMENT_ROOT    $document_root;\n        fastcgi_param   SCRIPT_NAME      $uri;\n        fastcgi_param   SCRIPT_FILENAME  /srv/markdown/md.php;\n        include fastcgi_params;\n    }\n}\n
\n

That's all there is to it! Reload Nginx and you should be all set.

\n

Optional improvements

\n

Nicer error pages

\n

Check this out. Who wouldn't want that over this?! Luckily, this is super easy to set up. All you need to do is add the following to your server block.

\n
proxy_intercept_errors on;\nerror_page 403 /404.html;\nerror_page 404 /404.html;\nlocation = /404.html {\n    root /srv/www;\n}\n
\n

Finally, put whatever 404 page you want in /srv/www/404.html and Nginx will serve it whenever a 404 or 403 error occurrs, which S3-compatible endpoints tend to send when files are not found.

\n

Caching

\n

You will probably want to set up some kind of caching in order to reduce latency. Add the following to /etc/nginx/nginx.conf:

\n
proxy_cache_path /srv/www/spaces/cache  levels=1:2    keys_zone=SPACES:50m         inactive=168h  max_size=10g;\n
\n

You can set the path to whatever you want, of course. Just make sure that www-data has write access to it.

\n

Then, like we did with the common S3 directives, save the following in /etc/nginx/snippets/spaces_cache.conf.

\n
add_header X-Cache-Status $upstream_cache_status;\nproxy_cache SPACES;\nproxy_cache_valid 1d;\nproxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504;\n
\n

This will cache files for one day. Because every file has a random string appended to it, I don't really worry about the cache period being that long, but you might want to lower it.

\n

Now, go back to your server block, and add the following to every block that has a proxy_pass directive to Spaces.

\n
include snippets/spaces_cache.conf;\n
\n

SSL

\n

Ok, this is not so optional really. Please set up SSL. With Let's Encrypt, you really have no excuse not to do so.

\n\n

Short URLs

\n

I use my own short URL for files, kmln.sr. This lets me have neat links like https://kmln.sr/qa6. klein is a super easy-to-set-up URL shortener that I wrote and use. The README should provide you with all the necessary documentation. Once that's set up, you can configure Dropshare to use it under the Uploads tab like so:

\n
\"screenshot\"
\n

Useful Resources

\n\n","published_at":"2017-10-02T15:51:20.000+03:00","slug":"publishing-screenshots-screencasts-and-other-files-on-digitalocean-spaces","tags":[],"plaintext":"A couple of weeks ago at DigitalOcean, we introduced Spaces\n[https://www.digitalocean.com/products/object-storage/]—beautifully simple and\nreliable object storage. With Spaces, you can host and serve static files\nwithout having to worry about security, scaling, disk space, and all the fun\nthings that come with maintaining a server.\n\nSpaces is compatible with the S3 API. This is important because it makes Spaces\nfully available to use with a huge number of applications and tools right away.\nIn this post, I'll talk about how I'm using Dropshare [https://getdropsha.re/] \nto share screenshots, markdown files (rendered as HTML!), and all sorts of files\non a custom domain name backed by Spaces.\n\nFeatures\nThe most important part about this is that you are able to use your own domain\nname instead of bucket.region.digitaloceanspaces.com. But that's not all. In\naddition to simply serving static files, kmlnsr.me also renders markdown files\nas HTML, with the ability to view the raw Markdown source code. Take a look at \nthis file [https://kmlnsr.me/markdown_example-t5QY2X.md] for example. It's a\nMarkdown file that is rendered as HTML with styling, on the fly. You can view\nthe original Markdown code by appending /raw to the URL\n[https://kmlnsr.me/markdown_example-t5QY2X.md/raw]. Spaces does not support that\nnatively, though. More on how that works in a bit.\n\nSetup\nUploading files\nDropshare [https://getdropsha.re/] uses the S3 API to upload files to Spaces.\nIt supports custom S3 endpoints natively, so it's very easy to configure. Simply\ncreate a new connection with the following configuration:\n\nYou might also want to replace the default screenshot hotkey to use Dropshare.\nFirst, disable the default hotkey in System\nPreferences→Keyboard→Shortcuts→Screen Shots. Then, configure Dropshare to use\nthat shortcut in the Screenshots tab.\n\nFor reference, here's Dropshare documentation on setting up DigitalOcean Spaces\nconnections:\n\n * How to set up a DigitalOcean Spaces connection with Dropshare\n [https://dropshare.zendesk.com/hc/en-us/articles/115002177973-How-to-set-up-a-DigitalOcean-Spaces-connection]\n\nServing files\nNow this diagram is a bit more complex than the previous one, so let's break it\ndown piece by piece.\n\n 1. A browser makes a request to a file on kmlnsr.me (like the screenshot\n above!). The request goes to a Droplet with nginx running on it.\n \n \n 2. Nginx looks at the URL and checks if it is a Markdown file.\n \n a. If it is not a Markdown file, it proxies the request to Spaces, fetching\n the file and serving it as is to the browser.\n \n b. If it is a Markdown file, it fetches the file from Spaces, passes it to\n a markdown renderer, and then serves the HTML version to the browser.\n \n \n\nNginx Configuration\nLet's build the configuration bit by bit, starting with a basic one:\n\nserver {\n listen 80;\n\n server_name kmlnsr.me;\n root /dev/null;\n \n location = / {\n return 302 https://kamal.io;\n }\n}\n\n\nRight now, it doesn't do much. http://kmlnsr.me/ redirects to https://kamal.io,\nand everything else returns a 404 Not Found error.\n\nServing static files\nThe idea behind serving static files is fairly simple. Proxy requests to \nhttps://kmlnsr.nyc3.digitaloceanspaces.com without any post-processing.\n\nBecause we will set up a couple of location blocks, let's extract the common\ndirectives into their own file that we can simply include wherever we need it. I\nsaved it in /etc/nginx/snippets/spaces_proxy.conf.\n\nproxy_http_version 1.1;\nproxy_set_header Authorization '';\nproxy_hide_header x-amz-id-2;\nproxy_hide_header x-amz-request-id;\nproxy_hide_header Set-Cookie;\nproxy_ignore_headers \"Set-Cookie\";\nproxy_intercept_errors on;\n\n\nNow, let's serve some static files! location / matches all requests—except / \nitself as it is matched by a previous location block.\n\nlocation / {\n include snippets/spaces_proxy.conf;\n \n proxy_set_header Host kmlnsr.nyc3.digitaloceanspaces.com;\n proxy_pass https://kmlnsr.nyc3.digitaloceanspaces.com;\n}\n\n\nThe proxy_set_header directive is important. Without it, Spaces would receive \nkmlnsr.me as the Host and wouldn't know what bucket it is, as it doesn't\nunderstand custom domain names.\n\nWith this config, everything should work as if you are accessing \nhttps://YOUR_SPACE.nyc3.digitaloceanspaces.com directly.\n\nServing Markdown files\nThere are two parts to this feature:\n\n 1. Serving the raw Markdown file as-is\n 2. Taking that, rendering it, and serving it as HTML\n\nServing raw Markdown\nRight now, if you browse to /markdown_file.md/raw, you would get a 404 Not Found \n error. This is expected, because you are trying to access a file named raw in\nthe directory /markdown_file.md.\n\nIn order to fix that, we need to modify the URL that is passed to Spaces,\nremoving the /raw bit. This can easily be done with a bit of RegEx magic.\n\nlocation ~ ^/(.*)\\.md/raw$ {\n include snippets/spaces_proxy.conf;\n \n add_header Content-Type text/plain;\n proxy_set_header Host kmlnsr.nyc3.digitaloceanspaces.com;\n proxy_pass https://kmlnsr.nyc3.digitaloceanspaces.com/$1.md;\n}\n\n\nThe URI pattern ^/(.*)\\.md/raw$ matches requests in the format of \n*/markdown_file.md/raw, capturing the path to the Markdown file in the process.\nOnce we have that, we simply set the URL that is sent to Spaces to \n*/markdown_file.md.\n\nThis way, we get the file's content, and serve it back to the browser. Setting\nthe Content-Type header to text/plain asks browsers to display the file as\nplain text instead of downloading it.\n\nRendering Markdown as beautiful HTML\nI've had this set up for years, so it uses a simple PHP script to do the hard\nwork. You can download the source code here [https://kmln.sr/6cZ], hosted on\nSpaces of course :)\n\nIn that archive, you will find two files:\n\n 1. markdown.php this is the Markdown library that I'm using. It's a very old\n version, but it has been working well so why bother replacing it ¯\\_(ツ)_/¯\n 2. md.php this is the file that Nginx runs. It's not super complex, but it\n basically requests the source (using the /raw URL), parses it using the\n Markdown library, and serves it along with a basic CSS stylesheet.\n Unfortunately, I don't remember where I got the stylesheet from.\n\nYou will need to adjust a couple of things in md.php. First, set the $url \nvariable (line 5) to your own hostname. Then, copy the stylesheet to your Space\nand update the URL on line 21.\n\nlocation ~ \\.md$ {\n fastcgi_pass unix:/var/run/php5-fpm.sock;\n fastcgi_param DOCUMENT_ROOT $document_root;\n fastcgi_param SCRIPT_NAME $uri;\n fastcgi_param SCRIPT_FILENAME /srv/markdown/md.php;\n include fastcgi_params;\n}\n\n\nThis is a basic php-fpm configuration. The most important part is that \nSCRIPT_FILENAME is fixed and always set to the md.php handler.\n\nPutting it all together\nOnce you're all done, the config file will look like this:\n\nserver {\n listen 80;\n\n server_name kmlnsr.me;\n root /dev/null;\n \n location = / {\n return 302 https://kamal.io;\n }\n \n location / {\n include snippets/spaces_proxy.conf;\n \n proxy_set_header Host kmlnsr.nyc3.digitaloceanspaces.com;\n proxy_pass https://kmlnsr.nyc3.digitaloceanspaces.com;\n }\n\n location ~ ^/(.*)\\.md/raw$ {\n include snippets/spaces_proxy.conf;\n \n add_header Content-Type text/plain;\n proxy_set_header Host kmlnsr.nyc3.digitaloceanspaces.com;\n proxy_pass https://kmlnsr.nyc3.digitaloceanspaces.com/$1.md;\n }\n\n location ~ \\.md$ {\n fastcgi_pass unix:/var/run/php5-fpm.sock;\n fastcgi_param DOCUMENT_ROOT $document_root;\n fastcgi_param SCRIPT_NAME $uri;\n fastcgi_param SCRIPT_FILENAME /srv/markdown/md.php;\n include fastcgi_params;\n }\n}\n\n\nThat's all there is to it! Reload Nginx and you should be all set.\n\nOptional improvements\nNicer error pages\nCheck this [https://kmlnsr.me/asdokasodij] out. Who wouldn't want that over \nthis [https://kmlnsr.nyc3.digitaloceanspaces.com/asdokasodij]?! Luckily, this is\nsuper easy to set up. All you need to do is add the following to your server\nblock.\n\nproxy_intercept_errors on;\nerror_page 403 /404.html;\nerror_page 404 /404.html;\nlocation = /404.html {\n root /srv/www;\n}\n\n\nFinally, put whatever 404 page you want in /srv/www/404.html and Nginx will\nserve it whenever a 404 or 403 error occurrs, which S3-compatible endpoints tend\nto send when files are not found.\n\nCaching\nYou will probably want to set up some kind of caching in order to reduce\nlatency. Add the following to /etc/nginx/nginx.conf:\n\nproxy_cache_path /srv/www/spaces/cache levels=1:2 keys_zone=SPACES:50m inactive=168h max_size=10g;\n\n\nYou can set the path to whatever you want, of course. Just make sure that \nwww-data has write access to it.\n\nThen, like we did with the common S3 directives, save the following in \n/etc/nginx/snippets/spaces_cache.conf.\n\nadd_header X-Cache-Status $upstream_cache_status;\nproxy_cache SPACES;\nproxy_cache_valid 1d;\nproxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504;\n\n\nThis will cache files for one day. Because every file has a random string\nappended to it, I don't really worry about the cache period being that long, but\nyou might want to lower it.\n\nNow, go back to your server block, and add the following to every block that has\na proxy_pass directive to Spaces.\n\ninclude snippets/spaces_cache.conf;\n\n\nSSL\nOk, this is not so optional really. Please set up SSL. With Let's Encrypt\n[https://letsencrypt.org/], you really have no excuse not to do so.\n\n * How To Secure Nginx with Let's Encrypt on Ubuntu 16.04\n [https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-16-04]\n .\n\nShort URLs\nI use my own short URL for files, kmln.sr. This lets me have neat links like \nhttps://kmln.sr/qa6. klein [https://kmln.sr/klein] is a super easy-to-set-up\nURL shortener that I wrote and use. The README should provide you with all the\nnecessary documentation. Once that's set up, you can configure Dropshare to use\nit under the Uploads tab like so:\n\nUseful Resources\n * How to set up a DigitalOcean Spaces connection with Dropshare\n [https://dropshare.zendesk.com/hc/en-us/articles/115002177973-How-to-set-up-a-DigitalOcean-Spaces-connection]\n * An Introduction to DigitalOcean Spaces\n [https://www.digitalocean.com/community/tutorials/an-introduction-to-digitalocean-spaces]\n * Understanding Nginx HTTP Proxying, Load Balancing, Buffering, and Caching\n [https://www.digitalocean.com/community/tutorials/understanding-nginx-http-proxying-load-balancing-buffering-and-caching]\n * How To Secure Nginx with Let's Encrypt on Ubuntu 16.04\n [https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-16-04]\n .\n * Object Storage tutorials\n [https://www.digitalocean.com/community/tags/object-storage?type=tutorials]","meta_description":"A couple of weeks ago at DigitalOcean, we introduced Spaces—beautifully simple and reliable object storage. With Spaces, you can host and serve static files without having to worry about security, scaling, disk space, and all the fun things that come with maintaining a server."}}]}},"pageContext":{"slug":"publishing-screenshots-screencasts-and-other-files-on-digitalocean-spaces","prev":"check-how-many-saltstack-minions-are-connected","next":"exporting-email-threads-from-gmail-into-csv-file"}},"staticQueryHashes":["3649515864"]}