I've got streams being set properly for each category. The last bit is to clean up the content automatically and rewrite / swap out image tags. Particularly photos posted with Sunlit. Posts made with Sunlit are displayed as an <a> tag (which links to your original image with a "-scaled" suffix) and an <img> tag with a source that proxies through micro.blog.
It also has an attachment of the second photo, which I am automatically inserting into the post.
I need to extract all <a> tags, detect if their href attribute has a "-scaled" in it, and strip that. Then I can look in my database for that attachment entry in my db and rewrite the tag as a Tanzawa image insert. Shouldn't take too long. Maybe tomorrow.
There's light at the end of the tunnel. I can import all of my post content, including check meta-data, bookmark urls – everything. The only remaining tasks is to build a custom 404 handler that will redirect visits to the old Wordpress urls to their new Tanzawa permalinks and to use the configured Category -> Stream mapping record.
I’ve got all of my extractors written. Next up is actually importing the content. In my import I’m also going to automate cleaning up some of the markup.
- Removing link wrappers around images. I.e. images posted from Sunlit wrap all images with an a tag. I want to strip that.
- Rewriting all attachment links to their new Tanzawa permalink.
- Rewriting all internal links to their new Tanzawa permalink.
I have a few pages on blog. I’m not sure I want to support pages yet (at least not in such a free form). I could import them as posts, so the content moves over. But instead I think I’m going to move them to my wiki instead.
There's still a slog ahead for importing posts, but it seems manageable. I wrote a bunch of utility functions (with tests) to extract and normalize individual fields of data from a post.
The idea being, once I can extract the data easily, I should be able construct my records by simply calling each function (more or less).
Migrating comments is going to be tricky as I only support webmentions in Tanzawa and not all comments on my blog originate from webmentions. I think I'll probably just not import comments/webmentions until after I migrate my blog to Tanzawa.
The last thing I need to do is import individual post content. Maybe it's because I'm not building fun features, but this last Wordpress import feels like such a slog.
I made a fun hack for importing images. I'm using (part) of the Hotwire stack for the dynamic portions of Tanzawa. Most dynamic web applications today use client side rendering, which means the server sends a json data structure and your browser has code/templates/logic to instruct it how to turn it into html for display. Hotwire is "html over the wire", so all of your logic and rendering happens on the server and the browser just displays the result.
Turbo also support lazy loading. Which means that it's not going to load the frame until it shows up on the page. Which means I can import all of my images by just scrolling down the page.
I broke the chain with a weekend off after about 3 months of working on Tanzawa a bit each and everyday. Today I'm back at it and I made a small api that imports images from Wordpress. Tomorrow I should be able to build a small interface that'll loop through the attachments and automatically download them.
Managed to get the category to stream and post kind to (tanzawa) post kind mappings working. I also got the attachment import records saving properly.
My basic plan for importing attachments is as follows. Each photo in Wordpress is exported as a post with the post type as "attachment". The guid for the item is the url for the attached file. So, I've created a record has a foreign key to the originating wordpress import record, the post guid, my own uuid, and a nullable foreign key to the resulting Tanzawa file attachment.
Once the file has been imported, I'll have a Tanzawa file attachment set so I'll easily be able to pick up where I left off.
Also since I'm keeping the originally uploaded Wordpress export file around and references between Tanzawa data and imported data, as I add features and capabilities to Tanzawa, I'll have the option to go back and pull in meta-data from Wordpress that I skipped on the initial import.
I've got the base interface for mapping categories to streams worked out. Adding a new stream directs you to the django admin, which isn't ideal from a user perspective, I kinda like it because modifying system data should be different.
Have got the base upload form working. After uploading the file it's automatically creating placeholder records for all post formats (Wordpress' build in Post Kind), Categories, and Post Kinds.
But thinking more about the actual worflow, I think it will be better not automatically create those records and truly split it separate steps. So after uploading the file you're taken to a list page with a list of all uploaded Wordpress files.
Next to each filename there will be five buttons: "Set Category Mapping", "Set Post Format Mapping" , "Set Post Kind Mapping" (if found), "Import Media", and "Import Posts". The "Import Posts" button will be disabled until mapping has been setup and media has been imported.
Uploading the file will automatically redirect you to the "Set Category Mapping" page, but if you leave the process midway through you'll be able to pickup where you left off.
The Small Web is for people (not startups, enterprises, or governments). It is also made by people and small, independent organisations (not startups, enterprises, or governments). On the Small Web, you (and only you) own and control your own home (or homes).This is exactly what motivates me to work and build Tanzawa. The world needs a smaller web focused on people. The Small Tech principles are also bang on.
Getting the base models used for importing Wordpress posts into Tanzawa built has made the task feel a bit less daunting. There's a clear path forward.
The next big challenge for Tanzawa, and the last thing required for me to switch to it, is import my data from my existing Wordpress blog. There's 4 major parts to this challange:
1. Parsing the Wordpress export XML file
2. Figuring out how to map Wordpress posts to Tanzawa posts
3. Downloading and importing media
4. Rewriting existing posts to use these new asset urls and fix links.
The first step is the easiest. I've figured out the basics of it yesterday using Beautiful Soup, but will require more exploration of the various posts before I can decide how to properly map data.
The other steps are managable, but wrapped up in a 5th challenge – managing the entire import process itself. Initially I had planned on just making a command line import tool. Run the command and it does its best to import everything. But telling Tanzawa how to map categories to streams would entail complex command parameters, which I wouldn't want to use myself, let alone inflict on others.
Rather, I need a simple web interface and database tables that will let me manage and monitor the process. The basic workflow I'm imagining is something like this:
1. User uploads Wordpress export file -> Tanzawa saves it into a blob in its database along with some basic meta information about it.
2. Tanzawa will create a mapping record for each category/post/post-kind found in the file. In step 2 users will see a list of their Wordpress categories with a dropdown next to each one with the stream it should map to (not mapping is also an option).
3. Tanzawa will also provision a record for each photo and post to import. This will include its planned final permanent url, as well its existing permanent url, and will be central when rewriting content.
4. The photo records will track not only urls, but also file download status, so we don't download photos twice. There should be a page where users can see a list of all photos to import, the status, and perhaps a button to retry if it's failed.
One tricky bit will be that Tanzawa doesn't support background tasks. Which means I can either introduce them (don't really want to) or I need to find a way to control entirely by the front-end. I think a little a small Stimulus controller on the photo list page that loops through each photo and call an import api should be sufficient.
5. Once the photos have been imported there should be a big button to publish all the changes. This will be button that will actually execute the entry creations.
6. After importing is complete, all of the old Wordpress urls should automatically redirect to their new Tanzawa permalink.
Throughout this process I'll likely find data that I (should) import that I don't have a way to handle in Tanzawa - and as such I may need to create features to handle them along the way.
Thinking about how large of a task importing Wordpress properly is a bit daunting. But if I just make a little progress each day, piece by piece, I'll complete it before I know it.
This past week was spent rounding out support for checkins.
I was so focused on getting locations functioning and out the door that forgot to include microformatted data. I've now included it along with the map and added tests to ensure I don't break it in the future.
Building locations helped me figure out the best pattern for adding 1-to-1 related data to an entry. A checkin is a location and a checkin record which is the name of the venue and a url for the venue. The one limitation I built in surrounding checkins is that they must be created via a micropub request and can only be updated via the admin interface.
While it's possible to integrate with Foursquare's places api and allow people to "checkin" using Tanzawa, it's a much better experience to use Swarm app and backfeed it.
Syndications is different from locations or checkins because rather than being a 1-to1 relationship, they're a 1-to-Many relationship. i.e. a single entry can have multiple syndication urls.
Supporting multiple syndication urls from micropub is straight forward, I can just iterate over the urls and save. However, the admin interface allows me to add, update, and delete records. In addition to the form itself, it also requires a (hidden) management form to manage the number of records and so forth.
Thankfully I was able to work the pattern out so going forward, adding any other 1-toMany data for a post should be much quicker.
Parsing the Wordpress XML file with feedparser strips all of the Wordpress specific data. But it looks like I can use BeautifulSoup (which I'm using elsewhere) to get what I need. The "xml" parser preserves the CData, so I can get the encoded data, too. Progress.
Shipped full support for checkins and syndication urls . Yay! 🎉 Next up is support for importing posts from Wordpress...😬
I got the web interface working for adding and removing of syndication urls. Nice and simple. Once I've got them marking up properly, I think they're ready to ship!
What’s cool about this: you can watch for mentions of whatever you want, and those come to you in the same app where your other feeds live.Following Twitter searches in NetNewsWire looks super handy. Will have to add the #IndieWeb hashtag once this is released.
Back to building some base infrastructure again. This time for adding / deleting syndication urls. Deleting is simple enough – it's just a (hidden) checkbox with a small stimulus controller setup to toggle its label text between "Remove" and "Undo".
Adding a secondary form also works. I output an empty formset form into my template then use a small action to copy it, replace the id and update the management form, 5 lines of code, half of which is variable definitions.
My goal is to be able to reuse this whenever I need to add multiple related records to a post without any modifications.
It's a little bittersweet — on the one hand it feels good as I know that I have a new solution but, on the other, it feels a little sad that the fruits of so much time and effort are now redundant.I feel this when I improve implementations all the time. Code reflects our best understanding of a given problem to a given solution at a given time. Requirements changing naturally means that the problem has also changed.
When writing code I try to remind myself that all is temporary and will be deleted or rewritten at some point, so I best not grow too fond of it.
Creating a slick checkin experience is out of the scope of a personal website, so you'll need to use Swarm (or such) to checkin and backfeed them into Tanzawa. However, I often find myself wanting to expand that check into a proper blog post – to tell a story about what happened. So you should be able to edit a checkin on your blog.
The edit checkin page is the same all other edit pages, except it has two fields at the top: Where did you go? and What's its url?
I also started working on being able to add syndication urls to posts. I've already added support via micropub to save syndication urls. This lets you add / remove them for non-micropub posts.
When you can eventually set your identities on other sites in Tanzawa, it might be fun to customize the placeholder text. For example you could get it to use your username or randomly use different sites e.g. twitter on the first load, flickr on the second load etc...
I just took a quick look at my server logs. And I see that there are some bots that are constantly crawling my site. If that would be search engines, feed readers etc. that wouldn’t be a problem. But this are crawlers by companies whose websites try to seI wonder if there isn't a list of these cralwers out there that you could then automatically block at the nginx/apache level. Even handling it at the app server level would discourage them from crawling your site too much.
Just a small update today – posts made with micropub respect the published datetime and I'm saving /showing syndication urls. I still need a proper interface for both of these in the admin. This is what syndication links will look like in a post.
With support for locations finished, I was able to add support for checkins in a couple of hours this morning. You can only post a checkin via micropub, but you can edit them like any other post locally.
There's three remaining "tidbits" to be completed before I can mark location/checkin support is complete: use the sent published date time in the micropub request as the posts' published datetime, store/display the syndication urls in the post meta, and confirm microformat my location data. This is what they look like – checkin is from January 2nd :-)
Along with my regular status posts, I'm going to try to make a weekly roundup post for Tanzawa. As this is more of an experiment at this time, I'm putting them in the "Articles", but I may add a weekly stream just for these posts.
I build and launched the ability to associate a location with an entry. Initially I had planned on limiting locations to check ins and statuses, but decided against building in an artificial limitation.
Location support is baked into the Tanzawa micropub endpoint as well as the RSS feeds. Posts that have a location associated with them will display the location after the author's name in the posts' byline. RSS feeds will append the location name ( or coordinates where there isn't an address) to the end of the post.
While adding the map to the public post views, I also I did some cleanup. I had originally planned on having a 3 column layout for Tanzawa: left navigation, middle content, right meta. But having it split into 3 columns felt unnatural. I removed the meta-data from the third column, though it still exists.
I also cleaned up the footer so it's stuck of the page without extending the view port beyond the natural max. Practically speaking it means that you'd always get a scrollbar even if the content length didn't warrant it. Ironic given that the footer text reads "Made with care". This text is also now styled to reduce emphasis.
Posts that belong to multiple streams will have their streams highlighted on the left. There's also a new "Home" link that takes you to the top of the site.
Working a bit on how to integrate location information into a post. Right now I've got the location name appearing in the byline. Some day, I imagine this will be a link to search page that'll show you all posts within a given radius of that area.
Initially I had planned to have the map display on the right hand side of the post, in the "meta" column. But that felt like it's separate from the post. Moving down into the footer and increasing it's size a bit has helped it feel a bit more "at home".
Sorted how locations will display on a post publicly today. I really like the automatic zoom as your mouse gets closer to the center.
I've settled in on the final mapping UI for Tanzawa and I'm super pleased with how it's turned out.
I realized that because addresses vary so much by country displaying ta full street address as you'd expect to see them on a letter is a) a huge problem all unto itself and b) far too much information to display.
Instead I opted to show the information that you really care about: city / state (prefecture), and country. It updates whenever your marker moves. The reset button lets your location to the initial value, allowing you to undo miss-clicks. The remove button lets you remove a location from a post entirely.
I also no longer set an initial marker in the map and zoomed out by about half. It's still centered on the Tanzawa mountains by default.
Managed to get reverse geocoding (looking up the address from for a given coordinate) working when you click the map. I've also hooked up the remove location button to...remove the location.
There's two last tweaks I need to make: 1) I'd like a reset button to reset your location form to it's initial state; 2) The address form is going to be hidden entirely and the values will be displayed as a single line of text below the map to emphasize that the address is "just meta".
When I think about what I want in a map on a blog, my needs are fairly basic: posts that have a location should show a map with an indicator where the post was made and if I'm unsure of the coordinates (a guarantee) , I should to be able to search and find it on a map.
If the location too new or uncommon, it may not show up. In that scenario finding the location on the map and selecting in manually isn't large ask.
While maps are a an important point of many posts: that new coffee shop I checked in at, the location of that cool bridge in a photo I shared, or that time status I posted looking out the window of the shinkansen – they're not central or even wanted in most posts.
Sharing a thought, a checkin, or a photo is the point.
One day there might be public facing features where maps play a prominent role and are the point of a post. But until then the big maps will be reserved for when you're authoring a post and can use the extra space to pan and zoom, and on the public side, they'll be smaller and out of the way – they're not the point.
Mapping is coming along nicely. Today I got markers to move by clicking the map, serializing changes to geo-json for saving, and integrated leaflet-geosearch.
Next I'd like to automatically fill in the address when available, but in order to do so, I must first normalize the Nominatim addressdetail response to match up with my fields.
After making it so I can update my location for an entry, I started to integrate Leaflet, rather than use the default GeoDjango OSM widget. It's my first time working with Leaflet, so it's a bit slow going.
I've managed to get basic display of the map working as well as clicking to move the point. I think if I can get the point to be powered by some GeoJSON in a hidden textarea, I should be able to once again set and update locations on entries.
I got locations saving to notes properly. Next up allowing you to update locations when editing an entry. When saving I initially ran into the following error:
spatalite no such function: lwgeom_version
The issue was that Django's PointField default projection is 4326, which is used for spherical references e.g. Google Earth. However, I'm displaying a flat service (an OSM Map), which uses a different projection 3857. I fixed this by changing my database to match my most common display.
from django.contrib.gis.db import models as geo_models ... point = geo_models.PointField(geography=True, srid=3857)
First steps in displaying location in the Tanzawa admin interface. Default point is Mount Tonodake in the Tanzawa mountain range. Still using the default OSM map included with GeoDjango.
I imagine this will be a Leaflet powered map in the end. I've got a couple of different ideas surrounding the location interface:
- 1. I'm probably not going to display the full address form and instead have a single unified search form + map when no location is set.
- 2. Once an address is set, I may show it in plain text below the map. Perhaps there with an "edit" link to manually override the address for whatever reason.
- 3. I'll probably do an initial release as-is (using OSM) and then focus one of my later sprints on really polishing the mapping interface.
Handling "related" data about an entry is proving to be a bit more difficult than anticipated. It could be the way I've chosen to validate data. For my micropub endpoint I'm using DRF for request validation and then transform and pass that data to the Django Form I use when posting with the admin interface. The Django Form is important because it's where I prepare the actual records that get saved into the database.
Handling this complex data with DRF in micropub feels natural – mostly because DRF Serializers handle nested data natively and microformat data is nested. Django Forms are made for regular forms (single level) and don't handle nested data.
Flattening all nested "related" data (like locations) and putting it into a single form isn't a good long-term solution. That said, I do flatten nested data for content as it's required to create an entry. However using both Forms (for base entry data) and DRF serializers (for related data) in both my admin views and micropub endpoint seems like a bad architecture as I'm mixing concerns.
Rather I think it's I need to start introducing Formsets for related data for my webforms. And then have my micropub endpoint continue doing what it's currently doing: sanity-check the request, transform it into format that matches my web form request, and then process as usual. This will also keep the data flowing a single direction:
micrpub request: Micropub -> DRF -> Form Input -> Form -> DB
admin request: Form Input -> Form -> DB
Started working on support for checkins (posted via backend micropub only) and adding locations to any post kind. I'm not sure how I'll expose adding a location to a post. One way would be get your current location. The other would be to let you just pick a point on a map. Or maybe default to your current location and let you adjust it?
Since my coordinates are all stored as a geo-django PointField, my hope is that it will allow me to create fun little apps on top of Tanzawa. "Make me a rollup of my checkins in Yokohama", "Show me my photos from Texas", and so forth.
As you may be able to tell from the previous post I just launched Bookmark support in Tanzawa. I'm contemplating if I shouldn't add a dedicated "bookmarks" page that just makes a long list of bookmarks, but for now it's good enough.
I realized one of the hardest parts about microformatting your html is making sure you haven't broken it in some subtle way. Mine were broken. I added some tests that verify a) I fixed them and b) they won't get broken again.
I've got bookmarks working - it's been mostly a copy/paste of the reply templates and generalizing the views to share as much as possible.
One thing I noticed was that while my extract method works with sites with proper microformat or schema data - it doesn't work for sites without either. So tomorrow I'll modify it to default to the page title and then update that value based on parsed meta-data.
Added support for replies to my micropub endpoint today. Super simple. Adding support for bookmarks shouldn't take but a day or two as it's mostly the same as replies. Switching to Tanzawa is getting closer and closer.
I got a good question from Adam (@TalAdam) about why I have Tanzawa on a subdomain, rather than my main domain. While the answer is simple it's a good opportunity to discuss my plans for Tanzawa in the mid-term future.
To answer Adam's question:
- 1. When I started Tanzawa I didn't have a domain (or even a name), so I started with a domain I owned.
- 2. I need to build up my minimal feature parity of my current blog before I can switch my main domain from Wordpress to Tanzawa.
What's remaining for my minimal feature parity? Only three parts: bookmarks/likes, checkins (w/ maps), and photo posts. Technically four - but photo posts are lower on the list. Once that's done I'll need to figure out how to migrate my data and create a huge redirect map for nginx. Perhaps by the end of the month?
Once I migrate my main site to Tanzawa what's left for this blog? I plan to redirect it somewhere on the tanzawa.blog domain. From there, I'll continuing using this blog as a development blog as I polish Tanzawa for a proper release that other people can use.
I started working on replies in Tanzawa. Introducing Turbo to add some dynamism to admin interface. It’s turned into a bigger rabbit hole than expected. I imagine most rails devs are familiar with the basics of Turbo (Links) but as a Django dev, things like Turbo aren’t included, so there’s a bit of a learning curve for me to implement it properly. That said – it’s coming along.Although I said it's coming along earlier today, I got it all working and I decided to ship despite replies not being implemented in my micropub endpoint yet.
One thing I'd like is to make responding to a tweet via Tanzawa as smooth as butter, so it's quick to tweet and reply from your own site. I imagine I can do that if I integrate with Bridgy.
Breaking the admin into TurboFrames and it feels like I'm rewriting the entire app. It's a good opportunity to refactor templates and views, but man do I feel like a hamster running on a wheel while I figure out the implementation patterns.
Reworked my template a bit and it seems that comments are now coming through!
I can now extract the reply data for microformat and schema.org schemas when I reply. Editing and RSS is also working nicely as well. I'm sending webmentions, but I think my template needs some tweaking as the comment is coming through.
Mixing Turbo (for replies) and regular page loads (for status/articles) is starting to show it's limits. I think after I get replies shipped, I need to step back and refactor my admin template to go full-in with Turbo.
Working on the display of replies. It's still a big work in progress - but progress none the less. The publish date / by-line work well for articles, but adding in the reply to URL makes it feel like there's too much information.
Perhaps it would be better to make the response title itself a link and hide the url?
I realized while I've been blogging about the development of Tanzawa, I haven't talked much about my overall goals for the system. This post will dive a bit into what my goals are for the system and how I envision it working.
My main goal for Tanzawa is to make a system that makes it enjoyable and easy to blog (be it micro, photo, articles, whatever) while maintaining ownership of your content. Yes, you can achieve this Wordpress, to a degree, but it's not made for it.
David Shanske and the other devs who've made the IndieWeb plugins for Wordpress have done fantastic work. It got me back into blogging and fighting for the open web. Tanzawa is my attempt to build upon the ideas in their work and push it forward.
Sustainability & Privacy
My other main goal for Tanzawa is to bring awareness and advocate for a sustainable web. Modern computer systems waste so many resources with poor design, unoptimized media, legacy formats - you name it. Tanzawa will always strive to have the lightest impact on your server, your network, and when rendered on your computer/phone.
To those ends Tanzawa uses minimal css styled with Tailwind - no large component frameworks. Image tags are written so browsers always choose the latest file format with the best compression so less data is transferred. What's more, we don't change file formats until it's first requested, saving your server from making and your disk from storing images that will never be served. We proudly use system fonts, avoiding megabytes of downloads.
A smaller, but still important goal for Tanzawa is to build a system that respects your privacy and the privacy of your visitors. We don't include any third-party libraries or scripts that track anything about you. We also strip all location and other exif from uploaded photos.
Own Your Data
One of the promises of the of POSSE and backfeeding is that you are in control. The current tooling works, but it feels too geeky. I either need to have a custom setup with a static site (which means writing Markdown and using a command line) or use Wordpress, and then you're stuck within the confines of Wordpress.
I want to make it simple to "tweet" from your blog. To backfeed from the silos into your blog. And I want to be able to remix this data together, to make new posts and pages, rather than having it locked in posts or behind an opaque API.
The recent resurgence of interest in blogging and RSS makes me hopeful that when Tanzawa is ready to general usage there's a chance it'll make a difference.
Still getting the hang of working with Turbo and Stimulus, so not much visible progress today. However, I maanged to automate submitting the reply-to url and refactored the design of the reply page.
I've got a basic Turbo-powered response form built. Using extruct to extract the meta schemas from the page is really simple.
Nothing too big to show yet, but I've started integrating Turbo for use with replies.
The basic idea is you paste in the URL you're replying to, Tanzawa will fetch the page and extract the meta information and display it in a form along with an area for you to input your response.
You always notice small bugs once you ship things. Fixed the order of recent posts in the dashboard and only opening webmentions / drafts if there's actual content to display.
With support for articles shipped in Tanzawa (this is the first one! 🎉 ) I'm taking a day off coding and doing a day of thinking about the next post kind(s) I want in Tanzawa: bookmarks & replies.
Why group them? Well, they're quite similar in my mind. A "reply" is a blog post that is a reply to some other page or post on the internet. On my current Wordpress site they look like a note with a link to site and maybe an excerpt for which we're replying.
Bookmarks look quite similar. Infact they're exactly the same, except I opted to put the link emoji for bookmarks.
Just looking at the two different post kinds the only difference is in the webmention that's sent. In replies there's a `u-in-reply-to` while in a bookmark there isn't. Part of the reason why they're the same is because how I treat bookmarks.
When I bookmark something publicly on my blog I've write a note as to why I'm bookmarking it. It's less of a reply and more of an aside. However, reflecting on this behavior while writing this post, I think I've been using bookmarks in this manner because I've been using Wordpress. The interface for posting bookmarks and replies (and statuses and articles) is the same and so you're encouraged to treat them the same.
Differentiating Replies and Bookmarks in Tanzawa
Both bookmarks and replies will need a field for inputting the url that we're bookmarking or replying too. And both kinds will need to dynamically load the page, extract title / author / summary information. And in both types users should be able to correct the extracted information.
I think the major difference must be made in how they're displayed in a list. Bookmarks should be displayed completely differently than other post kinds. They should only display the page title / link / date bookmarked / post permalink. Each bookmark's detail page can show the extra meta information: a note, an excerpt, and so forth.
Replies will display much like status i.e. there's no post title. But it will begin with a link and excerpt for the reply to setup the context for the post followed by our note. RSS feeds for bookmarks and replies should be the same: linked site meta info followed by a note.
My bet is that bookmarks and replies display much differently in your streams on the site that even though the publishing interface may be similar, your usage will change.
Rounding out the dashboard a bit. I've added a lot of all draft posts and the 5 most recent published posts. Webmentions only display if there's less than 3. If there's more than 3, they'll be hidden by default like below.
Drafts post kind is also at half-opacity to help visually separate them from published posts. I'm not sure if that's extra visual noise or not...what do you think?
Today I started working on cleaning up the post list interface for the admin. As I can now post articles or statuses it doesn't make sense to show a menu on the left that says "Status" when I really want to show all posts.
Each post shows an icon indicating the type of post that it is, the name (if it has one), date created, a preview of the content, and links to edit or view the post. I also got to re-use my post dropdown menu component yesterday.
Just a couple more tweaks and I think I can release articles.
I think it still needs a little fine-tuning with the display timing. Seeing the toggle when a user goes to click New Status is jarring. Likewise closing immediately when no longer hovering is a source of frustration. So adding a short delay before performing either action would likely make it a bit easier to use.
The other side-benefit of starting to use stimulus is that it's helping me break up my templates into more reusable and logical components.
I've started working articles with Tanzawa. Fundamentally they're the same as notes, except they have a name. Authoring an article looks much the same as a status.
Viewing an article on Tanzawa adds a title to the top. I also revisited the "byline" and made it italic. I've added light gray border to the footer of the post so you can visually tell when a post ends. The main post list view also has been updated to use this same color.
Once I add some tests to confirm that micropub is assigning post kinds properly I think I can ship article support. Once shipped, I may modify my publishing schedule from "at least one post a day everyday" to a weekly round up plus a status note or two as I have work to share.
The first step to being able to add new articles or bookmarks is to provide a way to navigate to the "new article" page. Here's what it's looking like so far.
Another week another feature coded and launched. This week I managed to get streams built and launched. You can see them on the right. Clicking each stream will filter the posts by the streams you selected when writing your post.
Steams can also be set via micropub if the client just sends a matching slug in the "category" property. Streams have feeds accessible at /<slug>/feed. You can subscribe to just Tanzawa status updates with status feed. As I can only author statuses on the backend right now this feed will match the main feed.
The other visual change you'll notice on the site is more navigation items on the left. Clicking on them allows you to navigate between streams.
With Streams finished, the next step for Tanzawa is going to be to allow me to author the various kinds of posts that would appear in these streams.
For articles I need to add posts with titles (technically supported via micropub, but titles aren't visible anywhere). Replies are more complex as I need to capture the url we're replying to, grab / parse the page for author / summary data and let the user customize it. It's a big task, but it will also finally let me start integrating Turbo in the admin.
Feeling a bit tired today but not wanting to break the chain of progress, I managed to get just a little bit done today.
I implemented Stream RSS feeds. This allows readers to subscribe to just a specific stream, rather than the entire river. You could also use the stream feeds to selectively syndicate content elsewhere e.g. I want all my status and article posts to syndicate to micro.blog, but not my checkins.
When authoring a post you can now select which streams you'd like for them to appear in. As each stream can have different visibility settings we show them along with the stream in understandable terms.
Next up is adding feeds for each stream type and setting the stream on micropub requests.
I started work on implementing streams this morning. This is what it's looking like on the public side with the default streams. Feels nice to be working on user-facing features again.
No coding today. Planning and thinking about how to implement Streams, one of my core ideas behind Tanzawa.
Streams will help you categorize and posts and blogs. You have a running stream, a status stream, a checkins steam. Each stream is independent of each other and each stream flows into the main Tanzawa stream.
Posts of any kind can appear in any stream. Streams will be listed on the left and have their own feed. In micropub parlance these would be tags or categories.
Some streams may be unlisted streams, entirely off the map, hiding all contents from anyone that isn’t logged in.
Once streams are working, I can start modeling checkin and address data requirements and add support to micropub.
The other thing I’m starting to think about is how other people could start using Tanzawa. Initial setup a little involved as SQLite needs Geo extensions installed to work. A Docker container would be easiest to make it all work, but even that’s a hurdle to getting started.
I could provide hosting, but I’m not sure I want that kind of responsibility yet. One step at a time.
After confirming that the <html> tag inside the <figure> tag was causing the errors with Feedly I went and fixed all posts. Below is the script I ran in the django shell.
Happy to report that Tanzawa is once again producing valid rss.
from bs4 import BeautifulSoup from post.models import TPost for t_post in TPost.objects.all(): entry = t_post.ref_t_entry.first() soup = BeautifulSoup(entry.e_content, 'html.parser') for html in soup.find_all('html'): try: pic = html.find('picture').extract() except AttributeError: # no pic continue html.replace_with(pic) entry.e_content = str(soup) entry.save()
Also being able to back up and restore your site's database with a simple "cp db.sqlite3 db.sqlite3.bak" is soooooooo nice.
As of today, Tanzawa officially supports IndieAuth and Micropub. Micropub is still a work-in-progress, but it works for basic note and articles with photos. I also pushed the RSS fix so _new_ posts with images should display properly for Feedly users. By the way - this post was made with Quill.
Figured out (I think) the issue with my Feedly feeds and images. I use BeautifulSoup and Django templates to rewrite the image tags generated by Trix to be lazy loading and offer optimized formats on the backend.
picture = BeautifulSoup( render_to_string("trix/picture.html", context),"html5lib" )
The issue is that the the "html5lib" causes my <figure> tag to be wrapped in <html> and <body> tags. Browsers are smart enough to filter this out. But these other parsers are not.
Changing this line to use the "html.parser" prevents wrapping my template in html tags and should solve the issue.
picture = BeautifulSoup( render_to_string("trix/picture.html", context), 'html.parser' )
Made good progress on the embed based64 image -> file save in tanzawa -> content rewrite. Now working on my handling of json micropub requests.
This handler is a bit large. I can't wait for it to be working so I can refactor it down to something a bit more manageable.
Those that are familiar with micropub know that there's two ways to use the endpoint. One is as a regular json api and the other is a plain-old form submit endpoint. Almost all integrations these days are done with json apis, so the idea of support a form api seems antiquated at first.
But once you realize that any site can publish to your site, including file attachments(!), with a simple form on their website, it doesn't seem so antiquated after-all. It feels quite liberating. In fact, it may be my favorite part of micropub.
Currently regular json requests can create a post. And form requests with file attachments can also create a post. Tanzawa automatically inserts the photo into the post and does its exif stripping, lazy loading, and other image loading optimizations.
Testing with the Quill article interface - image attachments are sent in base64 encoded strings in the img tag. Extracting those images, saving them to disk, and rewriting the html will take me another day to complete.
Feedly is still mangling my posts. Besides the figure tag messing things up for feedly, another theory I have is that it doesn't like text content not wrapped in a <p> tag. Trix (the editor Tanzawa uses) wraps all content in a <div> tag and text is raw inside of it. I pushed an experiment to rewrite that <div> to a <p> tag in the RSS feed. Hopefully this fixes it.
I've noticed some errors when Feedly parses my Tanzawa RSS feed. Namely that it doesn't show any content when I include an image. I have no idea why it's stripping all text from the posts.
Other fixes to the RSS feed include removing a / from the guid and setting the permalink to false, since the guid was not a url. As the guid change, most reads will show old posts as new - sorry about that.
This mostly a test post to see if adding titles for my statuses (the first 128 characters of plain text) will make Feedly parse properly when I include an image.
If this doesn't work, I have a theory that Feedly may not like <figure> tags, so I may try to rewrite my html content to be simpler for rss feeds.
Micropub is going to be a larger task than anticipated. As such I think I'll roll it out incrementally, expanding the capabilities of the micropub endpoint as Tanzawa can support them natively.
Today I managed to get my first post from Quill to Tanzawa!
Added support for verifying and revoking tokens. One thing I like about API development vs regular page/form development is the inputs and outputs are clearly defined. You don't need to worry about things like layout or looks. It's either valid or it's not.
With the basic indie auth login flow working, the next couple of days will be dedicated of fully fleshing out apis to support revoking tokens (e.g. on logout ) and checking token validity. I added basic tests for the authentication -> access token call today, so I can be confident that while I make these changes, I won't break anything.
I also added Bearer token based authentication today, which I've plugged into my micropub endpoint stub. My goal is to have the full indieauth/micropub api ready for release by Friday, at least on a basic level.
Importing data from OwnYourSwarm like I want will require a few more models and planning because posts will need to support geo-coordinates and so forth.
Woohoo! First successful IndieAuth login with Tanzawa. 🎉🙌🏻
I got the IndieAuth authorization api working. Initially I had built it out using the rest_framework.authtoken. However I realized their models only allow one token per user. This would make integrating with multiple clients impossible, so I'm opting to generate / manage tokens myself.
I can almost complete a sign in with OwnYourSwarm. I think my token endpoint response is wrong, which is preventing it from completing. Hopefully tomorrow I can get the complete signin working so I can move on to the micropub endpoint.
I feel like I'm starting to break small bits here and there inadvertently . So perhaps before building out the micropub endpoint and potentially breaking more things, I should take a step back and start adding some unit tests for my apis. It'll cost me a day today, but save me days of headaches over the course of the project.
Got the start of the indieauth authorization flow working and the base models setup.
Just a small update today. I shipped the login screen I shared yesterday and planned out what I need to build to support IndieAuth. I think I can use DRF to help handle token generation and authentication using the token.
With webmentions working, next up I want to tackle support for check-ins. Currently I'm using OwnYourSwarm to backfeed my checkins to my main site and I'd like to continue doing so with Tanzawa.
Doing so requires that I add support for IndieAuth (so I can login using just my domain) and Micropub. I'm starting on the IndieAuth implementation, which means I need to actually start with a login page for users to authenticate (thus far I've been using the django admin).
This is what it's looking like so far. Something feels off in the design of it, but I can't quite place my finger on what it is.
Yesterday when I shipped sending webmentions I ran into an error that didn't occur in development. Posting the webmention would work, but my response would timeout when trying to save content.
When updating a post you're supposed to send webmentions, update your content, then send webmentions again. As I'm trying to keep server requirements as simple as possible I'm doing all of the sending inline.
Looking at my logs what appeared to be happening was everything would just lock, then once my timeout occurred, I'd see my initial post request to Tanzawa come through, the timeout, and then Wordpress making a request to Tanzawa to retrieve it's mention.
This was happening because gunicorn only has 2 workers by default, which wasn't enough to handle processing a long request simultaneously with an incoming request. Increasing the workers from 2 to 4 solved the issue.
Today marks 1 month since I got the first instance of Tanzawa live. 🎉
Today I shipped support for sending webmentions. So now when I link to a post, Tanzawa will send webmentions. It's all done inline, so there's now a slow-down when saving. I should do it in the background, but I'm not sure I want to introduce redis/celery and all that complexity quite yet (or ever?).
I have a table you can view in the admin where you can see which posts sent which webmentions and if they were successful or not.
I also shipped a small update to webmention receiving. When an existing webmention is updated, I now also update how the webmention is displayed. In the case of an update, the comment must be re-moderated.
The final small "quality-of-life" update is when I save a post I now show a link in the success message to view the post.
I fixed my webmention receive implementation - I forgot to include the <link> tag in the head of my base template. With this fix in place comments on my Tanzawa micro.blog posts should be start to be sent here as well.
I also started on the other half of the webmentions: sending. Thus far I'm just trying to keep it simple and am using webmention-tools. I may switch over to ronkyuu as it seems better maintained, but it's good enough for now.
I successfully sent my first webmention from my local environment to my blog. Yay! I think I still need another day or so before I can call it ready for the web.
Mostly bug fixes today. Colin let me know about a bug how Tanzawa was interpreting webmentions from his blog. The basic issues was that Colin keeps all posts for the day on a single page and uses an anchor tag to link to the different posts e.g. blog.php?date=2021-01-28#p4 . So a single webmention request will include all posts for that day.
The mf2py-utils library I'm using to classify the microformated data as a comment works under the assumption that each post will have its own webmention. So when he linked to Tanzawa as post number two, Tanzawa showed the comment for post number one. It was an easy fix to make sure that the comment interpreting function filters by the target url as well.
The other minor change is I've added the interaction count to the status list page as well. I haven't styled it yet, but I imagine it will become a brown circle. This makes it easier for me to see which posts have comments/likes and so forth.
I'm on a roll with webmentions today. While I initially planned to hold off on displaying them publicly until a while later, since I figured out a good design, I decided to ship them in detail pages as well. They're hidden by default with a badge indicating the number of interactions. All mentions must be 👍ed before they appear. Pages without interactions do not show anything.
One thing I'm not entirely happy about is that by using the object tag to allow me to detect broken images is I lose the ability to lazy load them. i.e. img supports loading="lazy" and object does not. A waste of bandwidth and resources.
It's not a huge issue as icons are small. Once I start integrating turbo I plan to to lazy load the webmentions as a whole, so no content is event sent until requested.
I've also found a bug (or what I consider a bug) somewhere in the webmention stack (either in mf2py or in the webmention test tool, I think). The basic issue is emoji and other unicode characters come across as escaped unicode like \xf0\x9f\x98\xa2, Cleaning it the response body with the excellent ftfy (fixes text for you) takes care of the issue for now, but it feels like that should be unnecessary.
Figured it out. My handler wasn't running because I forgot to set the default app config in my __init__.py. One hotfix later and now we're cookin' with gas.
Shipped the new webmention display in Tanzawa 🎉. There is one bug. When django-webmention registers a model, I have a signal listening to register my own TWebmention record to store parsed data, moderation data and so forth.
Testing locally it worked fine, but my test webmention didn't seem to register it. Here's what it's looking like in prod:
Refactored the design of the webmentions this morning. I added the number of un-reviewed webmentions at the top and put them in a details/summary tag. The author name links to their homepage. Below that is a link to the webmention source displaying only the domain.
Reviews are handled via a big thumbs up or thumbs down on the far right. This bit isn't hooked up yet, but it will be soon enough. I also added a button to quickly take you to the new status page.
Next I need to add an api / view to handling the approval / disapproval click. And a menu item on the left to view all webmentions would be handy as well.
I started working on webmention displays for the dashboard in Tanzawa. Preview 👇:
Content is plain text and limited to 140 characters (after which it will truncate nicely). There's a link to review it on the left and the source of the webmention is also linked. The webmention kind is displayed on the bottom right if it can be determined.
Using the html5 <object> tag I can handle broken images or no author photo quite easily.
Come to think of it...these could be used as the base for displaying the interactions in a post as well...
Thank you to Colin for replying and sending Tanzawa its first non-test webmention.
Today I fixed some layout bugs and started working on a home / dashboard screen for the publishing side of Tanzawa. For starters it's going to display just the last 5 posts / webmentions and a button to write a new status.
Currently each wbmention shows it's review status (👍 or 👉), source url, and a link to the django admin to review (or delete) it. Instead of the source url (as that's not super helpful) I plan to show who it's from, the kind of webmention, and a link to view it. Eventually I'd like to normalize this data into the database, so I don't need to parse html on each page load, but I'm still working on the table schema in my head.
Once I get the dashboard "working", I think it's time to expand to other post types. Articles would be easiest - as it's just a status with a title. It will also force me to normalize sidemenus and post listing logic. After that I'd like to support checkins via a micropub endpoint so I can get my swarm checkins from OwnYourSwarm.
I've got webmentions working with the great django-webmention library. I'm parsing mention's microformats using mf2py and mf2util. It's not too difficult. You can send a webmention and I'll receive it. They just won't show on the page. And that's because I'm not sure how I want them to display.
For example a "like" doesn't necessarily have an author name, just a url. Or even no author at all. How do I display these? Do I just not display them? What's the appropriate microformat html for this type? Rich content or plain content? What if the plain content is in markdown? And repeat for each different kind.
I wanted to add support for receiving webmentions, as that way I can see if someone's linking to the tanzawa development blog, but I'll I think I'll hold off on display until I've had a chance to think through the implementations more.
Today I merged RSS support (feed), properly microformatted statuses, and automatic plain text conversion of statuses. Next up is some basic site settings, so you can set things like site title / feed title and so forth.
Computer is repaired and I'm making progress on Tanzawa again. Today I've got the statuses rendering as microformats and made an author page. Next up will adding RSS support for statuses.
Today's feature is a non-user facing feature: allowing tanzawa to be configured using a .env file via django-environ. Beyond letting me keep production settings / paths / secret keys out of git or keeping a separate settings.py for production, it also let me enable secure session cookies. Secure cookies with some apache/nginx configuration allows the url generated when you upload an image use https instead of http, which keeps the padlock in your browser locked.
Today I fixed image generation on linux ( turns out image/webp isn't recognized by python mimetypes on linux yet? ). I've also added permalinks for all posts. This will allow me to link directly to posts manually, but also sets me up for syndication and webmentions.
Progress may be slowed next week as my laptop might be getting its battery replaced.
Dynamic image format conversion is working nicely. I added a new feature to the image processing this morning as well. Rather than just strip geo data, I'm removing all exif data.
However, it also occurred to me that you may want to display a subset of that data on your website e.g. which camera the photo was taken with etc... So I'm saving all exif data to json before stripping it from the file itself. This should allow you to expose only the data you want explicitly in your templates. 🙌🏻
Adopting new image formats (webp/avif/apng) in place for legacy formats (jpeg/png/gif) is an area where websites simultaneously increase quality and reduce transfer size. I've got dynamic file format conversion working locally so browsers that support the new formats will automatically get the smallest file possible and older clients fallback to legacy formats. Just needs a bit more massaging before I merge it.
Upload to working. Basic micropub media endpoint working.
File models and geospactial settings finished. upload handler coded but not tested yet.
Starting to implement file upload support after planning yesterday. SpatiaLite looks super cool, totally going to use it.
Planned out support for images and media uploads. It's a little complex, but it should provide a lot of flexibility going forward.
Next on the Tanzawa task list, I need to decide how I want to handle images and what the (minimum) image processing pipeline will look like.
Got my first version of Tanzawa up and running. Super excited.