Sites back. I reindexed every table although the only one that seemed to be affected by the issues was the comment table (it had duplicates I had to remove similar to what I did for the post and community tables before)

Site will slowly catch up in activities over the next couple hours so youll see posts, etc. that were made in the few hours it was down slowly come in

Sites back, communities that previously had their community page not vieweable are now viewable

An example of this is https://programming.dev/c/a11y

Basically what I did was before the maintenance exported every single post in the post table and use a js script to check for duplicate ap_ids (checking using sql itself didnt work due to broken indices so it was always missing some). Then I generated sql statements to remove duplicate posts that had higher ids than the other posts theyre a duplicate of. Ran all those during maintenance and since all duplicate ap_ids were removed I could now reindex the table to fix the broken indices

Did the same for communities after the posts (but also did some statements to move posts from one version of the community to the other)

Site might be slow for a tiny bit as it processes the 2 hour of activities that were generated while it was down

Sites back up

Got a lot of information about what it actually causing the issues which should help me automate some fixes for tomorrow + the weekend

There likely will be some more downtime tomorrow if I have time (if not itll be on saturday)

The issue is essentially a chain of issues with one needing to be fixed before I can fix the others. Some tables have duplicates entries which is preventing reindexing (since the indexes are broken). First one im going to try to fix is the post table.

I did some changes to remove a duplicate community to test which is https://programming.dev/c/[email protected] and ive managed to make it merged and now comments will show if you sort by comments in the community. Posts however still aren’t showing up due to the posts table having the issue with duplicates as well. (fixing posts first so I can set all of the posts to the correct community in an automated way later for the communities)

Was an issue with the upgrade script when upgrading to 0.19.5

Caused by the same db issue thats been causing issues

Yeah they appear in the feeds. So people primarily use it through sub and local sorts atm

old is still supported, what comment section are you trying to look at

Newer communities arent affected

it cant have the same slug, just for the idea of remaking

it doesnt fully break communities, you just cant look at the page for the community and new people cant sub (but they can still be posted and read from). humor has still been active with posts while broken

next downtime hopefully should be soon to do another round of fixes

Theoretically remaking the community is the exact same as moving it to another instance since new communities arent affected. Everythings been getting worked on openly in the support channel on discord/matrix so if anyone wants to help can help get things fixed there

Been getting worked on, last downtime fixed some of the issues but script still needs to be adjusted a bit since some communities break again a bit after being fixed

People interact with the site that dont have accounts on the site

Making it local only would lock them out of news and discussion about something they interact with

There was some maintenance to try to fix things but things ended up not being resolved so had to be rolled back and there should be another maintenance period in the future again

should be back

Been getting worked on and people have been figuring out how best to fix the issue

there are references which is why its a pain to fix and isnt just dropping duplicates

I havent been the one looking at the database but im assuming itll be a similar way to how the person table was fixed. Theres essentially duplicate entries for certain communities atm

Its been getting looked at and we have some more people that are being added on to help out as well

Theres been issues with the communities table since the update and it hasnt been fixed yet since there hasnt been time to do so. Same issue that hit the person table but that one was fixed after the db maintenance we had

Community table currently has issues. Should be fixed once theres time to do more database maintenance again but for now the posts can be accessed in the main feeds